I have a JSON output that I receive from cURL, I would like to pipe to jq to extract useful information and display the csv data in a terminal.
Sample JSON response:
{
"operation": "GET CLIENT SESSIONS",
"outcome": "SUCCESS",
"result": {
"Session": [
{
"sessionID": "E1B7190EF32AA9F6E63265ADBB88D1A9D6F4254457A0.production",
"requestID": "",
"sessionPoolID": "8LNIVzRTSo-jJV-CdPMP0Q",
"agentID": "",
"ablSessionID": "",
"lastAccessStr": "2022-04-01T10:36:14.745-0700",
"elapsedTimeMs": 1675892,
"sessionState": "AVAILABLE",
"requestState": "READY",
"sessionType": "SESSION_FREE",
"adapterType": "APSV",
"bound": false,
"clientConnInfo": null,
"agentConnInfo": null
},
{
"sessionID": "E1B7190EF32AA9F6E63265ADBB88D1A9D6F4254457A0.production",
"requestID": "",
"sessionPoolID": "8LNIVzRTSo-jJV-CdPMP0Q",
"agentID": "",
"ablSessionID": "",
"lastAccessStr": "2022-04-01T10:36:14.745-0700",
"elapsedTimeMs": 1675892,
"sessionState": "AVAILABLE",
"requestState": "READY",
"sessionType": "SESSION_FREE",
"adapterType": "APSV",
"bound": false,
"clientConnInfo": null,
"agentConnInfo": null
}
]
},
"errmsg": "",
"versionStr": "v1.0.0 ( 2021-10-29 )",
"versionNo": 1
}
I am able to extract information using this query:
jq '.result.Session[] | ([.lastAccessStr, .elapsedTimeMs, .clientConnInfo]) | #csv'
The problem I am running into, sometimes the response can have additional nested data for clientConnInfo and agentConnInfo as seen in the example below.
{
"operation": "GET CLIENT SESSIONS",
"outcome": "SUCCESS",
"result": {
"Session": [
{
"sessionID": "E1B7190EF32AA9F6E63265ADBB88D1A9D6F4254457A0.production",
"requestID": "",
"sessionPoolID": "8LNIVzRTSo-jJV-CdPMP0Q",
"agentID": "",
"ablSessionID": "",
"lastAccessStr": "2022-04-01T10:36:14.745-0700",
"elapsedTimeMs": 1675892,
"sessionState": "AVAILABLE",
"requestState": "READY",
"sessionType": "SESSION_FREE",
"adapterType": "APSV",
"bound": false,
"clientConnInfo": null,
"agentConnInfo": null
},
{
"sessionID": "26691913A73E55175D233F86D219B4AEFA4AD14AE9E4.production",
"requestID": "ROOT:a:000002a6",
"sessionPoolID": "8LNIVzRTSo-jJV-CdPMP0Q",
"agentID": "qsrLXAxsRRanJio6dYOC2Q",
"ablSessionID": "",
"lastAccessStr": "2022-04-01T11:04:08.902-0700",
"elapsedTimeMs": 1735,
"sessionState": "RESERVED",
"requestState": "RUNNING",
"sessionType": "SESSION_FREE",
"adapterType": "APSV",
"bound": false,
"clientConnInfo": {
"clientName": "xxxxx",
"requestID": "ROOT:a:000002a6",
"sessionID": "26691913A73E55175D233F86D219B4AEFA4AD14AE9E4.production",
"adapterType": "APSV",
"reqStartTimeStr": "2022-04-01T11:04:08.902-0700",
"elapsedTimeMs": 1735,
"executerThreadId": "thd-8",
"requestUrl": "xxxxx",
"requestProcedure": "xxxxx.p",
"httpSessionId": "26691913A73E55175D233F86D219B4AEFA4AD14AE9E4.production"
},
"agentConnInfo": {
"agentID": "qsrLXAxsRRanJio6dYOC2Q",
"connID": "AR2k7gnYSiKPCk2DEHpaSg",
"connPoolID": "v0Lh7XcITsOSfoF_RjXXig",
"state": "RESERVED",
"agentAddr": "xxxxx",
"localAddr": "xxxxx"
}
}
]
},
"errmsg": "",
"versionStr": "v1.0.0 ( 2021-10-29 )",
"versionNo": 1
}
If I try to use the same query, it dies with below error:
jq: error (at <stdin>:0): object ({"clientNam...) is not valid in a csv row
The desired output is to capture this:
.lastAccessStr
.elapsedTimeMs
.clientName (if available in output)
.requestID (if available in output)
.agentAddr (if available in output)
.localAddr (if available in output)
I have been trying to make this work using https://jqplay.org without any luck.
Can anyone give me some examples on how I would go about making this work?
User pmf has got it in the comments:
Use ? to ignore an error, and // to provide an alternative if the first one is null, false or inexistent. Define every column at question along the lines of (.clientConnInfo.clientName? // "none")
Related
I have the following JSON input:
{
"paging": {
"count": 0,
"total": 0,
"offset": 0,
"max": 0
},
"executions": [
{
"id": 5,
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br",
"status": "succeeded",
"project": "PROJETO",
"executionType": "scheduled",
"date-started": {
"unixtime": 1660793400012,
"date": "2022-08-18T03:30:00Z"
},
"date-ended": {
"unixtime": 1660793409694,
"date": "2022-08-18T03:30:09Z"
},
"job": {
"id": "cdkwednweoi-8745bjdf-kcjkjr8745",
"averageDuration": 0,
"name": "routine",
"group": "",
"project": "PROJECT",
"description": "",
"href": "https://localhost.com.br",
"permalink": "https://localhost.com.br"
},
"description": "runner",
"argstring": null,
"serverUUID": "jdnsdnasldnaje382nf5ubv",
"successfulNodes": [
"84jsk937nf"
]
}
]
}
First I want to select an array by a property name. And then I want to select an object of the array by the value of the propertyes.
Example of the desired informations on output:
"href"
"status"
"project"
"date-started":
"unixtime": 48298437239847,
"date": "2022-07-17"
"date-ended":
"unixtime": 48298437239847,
"date": "2022-07-17"
"job":
"name": "cleaner"
I knew how to get the firts values:
jq -r '.executions[] | [.href, .status, .project']
But the other ones I don't know how to do, I've tried with:
jq '.executions[] | with_entries( select(.value | has("date-started") ) )'
But it doesn't works.
Your first query produces a JSON array, so in this response, I'll assume it will suffice to produce an array of the eight values of interest in the order you've specified.
With your input, the following invocation produces the eight values as shown below:
jq '.executions[]
| [.href, .status, .project,
(."date-started" | (.unixtime, .date)),
(."date-ended" | (.unixtime, .date)),
.job.name]'
Output:
[
"https://localhost.com.br/rundeck/api/40/execution/2340",
"succeeded",
"PROJETO",
1660793400012,
"2022-08-18T03:30:00Z",
1660793409694,
"2022-08-18T03:30:09Z",
"proc_limpeza_saft"
]
I'm making an API GET call using Ruby - the call is made to a Learning Management System and returns the following JSON:
{
"id": 12345,
"body": null,
"url": null,
"grade": "75",
"score": 75,
"submitted_at": "2020-05-02T11:30:53Z",
"assignment_id": 9876,
"user_id": 1111,
"submission_type": "online_upload",
"workflow_state": "graded",
"grade_matches_current_submission": true,
"graded_at": "2017-06-05T08:47:49Z",
"grader_id": 2222,
"attempt": 1,
"cached_due_date": "2020-05-03T15:00:00Z",
"excused": false,
"late_policy_status": null,
"points_deducted": null,
"grading_period_id": null,
"late": false,
"missing": false,
"seconds_late": 0,
"entered_grade": "75",
"entered_score": 75,
"preview_url": "https://etcetc",
"turnitin_data": {
"attachment_33333": {
"status": "scored",
"object_id": "44444444",
"similarity_score": 0,
"web_overlap": 0,
"publication_overlap": 0,
"student_overlap": 0,
"state": "none"
}
},
"attachments": [
{
"id": 33333,
"uuid": "kjsdkjhsdfkhsfd",
"folder_id": 55555,
"display_name": "Submission.pdf",
"filename": "Submission.pdf",
"content-type": "application/pdf",
"url": "https://etcetc",
"size": 2668226,
"created_at": "2020-05-02T11:30:51Z",
"updated_at": "2020-06-06T15:01:46Z",
"unlock_at": null,
"locked": false,
"hidden": false,
"lock_at": null,
"hidden_for_user": false,
"thumbnail_url": null,
"modified_at": "2020-05-02T11:30:51Z",
"mime_class": "pdf",
"media_entry_id": null,
"locked_for_user": false,
"preview_url": "api/etcetc"
}
],
"submission_comments": [
{
"id": 99999,
"comment": "here’s a comment",
"author_id": 1,
"author_name": "Mickey Mouse",
"created_at": "2020-05-15T12:54:08Z",
"edited_at": null,
"avatar_path": "/images/users/1",
"author": {
"id": 1,
"display_name": " Mickey Mouse ",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
},
{
"id": 223344,
"comment": "another comment",
"author_id": 2,
"author_name": "Donald Duck",
"created_at": "2020-06-05T10:48:51Z",
"edited_at": null,
"avatar_path": "/images/users/2",
"author": {
"id": 2,
"display_name": "Donald Duck",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
}
]
}
I need to be able to retrieve specific values from "submission_comments", namely the values for "comment", "author_id" and "author_name". At the moment the best I can do is retrieve "submission_comments" as one big entity. Here's how I'm getting that far:
require 'typhoeus'
require 'link_header'
require 'json'
require 'csv'
the_url = 'https://etctetc'
token = 'mytoken'
api_endpoint = '/api/etc'
output_csv = 'C:\Users\me\Desktop\Ruby Canvas course\assignment_comments.csv'
CSV.open(output_csv, 'wb') do |csv|
csv << ["user_id", "TII", "marker"]
end
request_url = "#{the_url}#{api_endpoint}"
count = 0
more_data = true
while more_data
get_comments = Typhoeus::Request.new(
request_url,
method: :get,
headers: { authorization: "Bearer #{token}" }
)
get_comments.on_complete do |response|
#get next link
links = LinkHeader.parse(response.headers['link']).links
next_link = links.find { |link| link['rel'] == 'next' }
request_url = next_link.href if next_link
if next_link && "#{response.body}" != "[]"
more_data = true
else
more_data = false
end
if response.code == 200
data = JSON.parse(response.body)
data.each do |comments|
CSV.open(output_csv, 'a') do |csv|
csv << [comments['id'], comments['turnitin_data'], comments['submission_comments']]
end
end
else
puts "Something went wrong! Response code was #{response.code}"
end
end
get_comments.run
end
puts "Script done running"
I'm new to this (the ruby code is based on an exercise so I may not fully understand it)- any help/advice would be really appreciated!
EDIT: I should also note that this isn't the total JSON response I'm dealing with - this is just one of ten items that are returned
"submission_comments": [
{
"id": 99999,
}
]
the [] means it is array. {} means it is an object.
So you probably need to do something like this:
json["submission_comments"].first["id"]
or better iterate through it:
ids = json["submission_comments"].map{|comment| comment["id"]}
I'm able to get the variables you need if you can read the JSON file in as text, then use Ruby's JSON.parse(...) method on it. I think the main problem is that JSON uses null but Ruby hashes use nil. You could do a string replace or try something like this (I did not modify your JSON, only put it into a single quoted string):
json_text = '{
"id": 12345,
"body": null,
"url": null,
"grade": "75",
"score": 75,
"submitted_at": "2020-05-02T11:30:53Z",
"assignment_id": 9876,
"user_id": 1111,
"submission_type": "online_upload",
"workflow_state": "graded",
"grade_matches_current_submission": true,
"graded_at": "2017-06-05T08:47:49Z",
"grader_id": 2222,
"attempt": 1,
"cached_due_date": "2020-05-03T15:00:00Z",
"excused": false,
"late_policy_status": null,
"points_deducted": null,
"grading_period_id": null,
"late": false,
"missing": false,
"seconds_late": 0,
"entered_grade": "75",
"entered_score": 75,
"preview_url": "https://etcetc",
"turnitin_data": {
"attachment_33333": {
"status": "scored",
"object_id": "44444444",
"similarity_score": 0,
"web_overlap": 0,
"publication_overlap": 0,
"student_overlap": 0,
"state": "none"
}
},
"attachments": [
{
"id": 33333,
"uuid": "kjsdkjhsdfkhsfd",
"folder_id": 55555,
"display_name": "Submission.pdf",
"filename": "Submission.pdf",
"content-type": "application/pdf",
"url": "https://etcetc",
"size": 2668226,
"created_at": "2020-05-02T11:30:51Z",
"updated_at": "2020-06-06T15:01:46Z",
"unlock_at": null,
"locked": false,
"hidden": false,
"lock_at": null,
"hidden_for_user": false,
"thumbnail_url": null,
"modified_at": "2020-05-02T11:30:51Z",
"mime_class": "pdf",
"media_entry_id": null,
"locked_for_user": false,
"preview_url": "api/etcetc"
}
],
"submission_comments": [
{
"id": 99999,
"comment": "here’s a comment",
"author_id": 1,
"author_name": "Mickey Mouse",
"created_at": "2020-05-15T12:54:08Z",
"edited_at": null,
"avatar_path": "/images/users/1",
"author": {
"id": 1,
"display_name": " Mickey Mouse ",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
},
{
"id": 223344,
"comment": "another comment",
"author_id": 2,
"author_name": "Donald Duck",
"created_at": "2020-06-05T10:48:51Z",
"edited_at": null,
"avatar_path": "/images/users/2",
"author": {
"id": 2,
"display_name": "Donald Duck",
"avatar_image_url": "https://etcetc",
"html_url": "https://etcetc"
}
}
]
}'
Part I added:
ruby_hash = JSON.parse(json_text)
submission_comments = ruby_hash["submission_comments"]
submission_comments.each do |submission_comment|
comment = submission_comment["comment"]
author_id = submission_comment["author_id"]
author_name = submission_comment["author_name"]
puts "Comment: #{comment}, Author ID: #{author_id}, Author Name: #{author_name}\n\n"
end
Terminal Result:
=> Comment: here’s a comment, Author ID: 1, Author Name: Mickey Mouse
=> Comment: another comment, Author ID: 2, Author Name: Donald Duck
Edit: I added a jenky af one-liner version just for fun (presuming the json_text variable above is already initialized)
JSON.parse(json_text)["submission_comments"]
.map{|txt| puts(["comment","author_id","author_name"]
.map{|k| k.instance_eval{"#{upcase}: #{txt[to_s]}"}}.join(', '))}
COMMENT: here’s a comment, AUTHOR_ID: 1, AUTHOR_NAME: Mickey Mouse
COMMENT: another comment, AUTHOR_ID: 2, AUTHOR_NAME: Donald Duck
I would like to create invoices by my PHP script. But before I am able to program this I would like to understand the API from Debitoor.
In order to create an invoice you have to create a draft invoice first. You can find this here: /api/sales/draftinvoices/v3.
I have set all the requested parameters to use the the /api/sales/draftinvoices/v3 function however, I always get a 400 Bad Request response back.. The reason is the schema.
Can you guys tell me where my failure is?
I would guess is something with the "lines"... but I would say it's all correct...
This is my request:
Request as text:
{
"date": "2018-05-06",
"dueDate": "2018-05-13",
"notes": "Diese Rechnung ist vom Backend erstellt worden.",
"customerName": "Max Mustermann",
"customerAddress": "Mustermann Stra\u00dfe",
"customerCountry": "DE",
"currency": "EUR",
"languageCode": "de-DE",
"recargoTaxEnabled": false,
"sent": false,
"viewed": false,
"displayAsPaid": false,
"lines": {
"taxEnabled": "false",
"description": "Leistungszeitraum 06.04.2018 - 05.05.2018",
"taxRate": "1.19",
"productOrService": "product"
}
}
This is the response I get back:
As text:
{
"message": "Error validating against schema",
"id": "9905636b-fb65-41a0-8d25-4aa096d5347d",
"code": "schema",
"errors": [
{
"message": "is the wrong type",
"value": {
"taxEnabled": "false",
"description": "Leistungszeitraum 06.04.2018 - 05.05.2018",
"taxRate": "1.19",
"productOrService": "product"
},
"type": "array",
"property": "lines"
}
],
"body": {
"date": "2018-05-06",
"dueDate": "2018-05-13",
"notes": "Diese Rechnung ist vom Backend erstellt worden.",
"customerName": "Max Mustermann",
"customerAddress": "Mustermann Stra\u00dfe",
"customerCountry": "DE",
"currency": "EUR",
"languageCode": "de-DE",
"recargoTaxEnabled": false,
"sent": false,
"viewed": false,
"displayAsPaid": false,
"lines": {
"taxEnabled": "false",
"description": "Leistungszeitraum 06.04.2018 - 05.05.2018",
"taxRate": "1.19",
"productOrService": "product"
}
}
}
Kind regards and Thank You!
Thank you for using the Debitoor API
You are correct. Your problem is with the lines. Lines should be an array. You have only send a single object.
Another issue is that you provide taxEnabled: false and taxRate: 1.19. If you provide taxEnabled as false you should set the taxRate to 0. Remember that taxRate should be between 0 and 100 with a maximum of two decimals
Your request should look something like this instead:
{
"lines": [
{
"taxEnabled": false,
"description": "Leistungszeitraum 06.04.2018 - 05.05.2018",
"taxRate": 0,
"quantity": 1,
"unitNetPrice": 1,
"productOrService": "product"
}
]
Best regards,
Carsten
Mobile developer # Debitoor
I've used pymongo to connect to mongo replica set and print the status of replica set using json dump. I want to parse this output and display "name" and "stateStr" into a list or array for the user to be able to pick a particular host.Here is my json dump output:
{
{
"replSetGetStatus": {
"date": "2016-10-07T14:21:25",
"members": [
{
"_id": 0,
"health": 1.0,
"name": "xxxxxxxxxxx:27017",
"optime": null,
"optimeDate": "2016-10-07T13:50:11",
"self": true,
"state": 1,
"stateStr": "PRIMARY",
"uptime": 32521
},
{
"_id": 1,
"health": 1.0,
"lastHeartbeat": "2016-10-07T14:21:24",
"lastHeartbeatRecv": "2016-10-07T14:21:24",
"name": "xxxxxxxxxxxx:27017",
"optime": null,
"optimeDate": "2016-10-07T13:50:11",
"pingMs": 0,
"state": 2,
"stateStr": "SECONDARY",
"syncingTo": "xxxxxxxxxxxx:27017",
"uptime": 27297
},
{
"_id": 2,
"health": 1.0,
"lastHeartbeat": "2016-10-07T14:21:24",
"lastHeartbeatRecv": "2016-10-07T14:21:24",
"name": "xxxxxxxxxxxxx:27020",
"pingMs": 0,
"state": 7,
"stateStr": "ARBITER",
"uptime": 32517
}
],
"myState": 1,
"ok": 1.0,
"set": "replica1"
}
}
Please try below Javascript code. It worked for me.
use admin;
var result = rs.status();
var length = result.members.length;
for (var i=0;i<length;i++){
print ("Server Name-" +result.members[i].name);
print ("Server State-" +result.members[i].stateStr);
}
{
"statusCode": "000",
"statusMessage": "Record Successfully Fetched",
"dsStatusCode": "000",
"dsStatusMessage": "Record Successfully Fetched",
"businessInput": null,
"businessOutput": {
"systemCircleId": "2",
"category": [
{
"categoryId": "abcs",
"sys": "5ID",
"displayName": "National Roaming Recharge",
"packsList": [
{
"amount": "79",
"benefits": "dsdsdsds",
"packId": "1344",
"processingFees": "70.3",
"serviceTax": "8.7",
"validity": "30 Days",
"volume": "0.0",
"isTop5": "no",
"fileName": "null"
},
{
"amount": "188",
"benefits": "Roaming Tariff - Incoming Free, Outgoing local # 80p/min, STD #1.15Rs/min with Talk Time 120 in main A/c",
"packId": "1263",
"fess": "47.3",
"serviceTax": "20.7",
"validity": "28 Days",
"volume": "0.0",
"isTop5": "no",
"fileName": "null"
},
{
"amount": "306",
"benefits": "FTT 306 with Roaming Tariff - Incoming Free, Outgoing local # 80p/min, STD #1.15Rs/min",
"packId": "1290",
"processingFees": "0",
"serviceTax": "33.7",
"validity": "28 Days",
"volume": "0.0",
"isTop5": "no",
"fileName": "null"
}
]
}
]
}
}
I want to parse this json to filter packlist for each category id using angularjs
assign a variable to the JSON you have. and use scope.$eval on the variable
Example
var jsonVar = { "statusCode": "000",
"statusMessage": "Record Successfully Fetched",
"dsStatusCode": "000",
"dsStatusMessage": "Record Successfully Fetched",
"businessInput": null
}
scope.$eval(jsonVar) // this gives the object on which you can do the ng-repeat
if you still have problems. Try using JSON.stringify(jsonVar) and then perform a scope.$eval on the this.
var jsonString = JSON.stringify(jsonVar);
scope.$eval(jsonString);// This returns a object too