Related
i have a json file like this:
[
{
"DisplayName": "Title1",
"SignInName": null,
"RoleDefinitionName": "Name1",
"RoleDefinitionId": "1111",
"ObjectId": "0111",
"ObjectType": "ServicePrincipal",
"CanDelegate": false,
"Description": null,
"ConditionVersion": null,
"Condition": null
},
{
"DisplayName": "Title2",
"SignInName": null,
"RoleDefinitionName": "Name2",
"RoleDefinitionId": "1111",
"ObjectId": "02222",
"ObjectType": "ServicePrincipal",
"CanDelegate": false,
"Description": null,
"ConditionVersion": null,
"Condition": null
},
{
"DisplayName": "Title3",
"SignInName": null,
"RoleDefinitionName": "Name3",
"RoleDefinitionId": "33333",
"ObjectId": "03333",
"ObjectType": "ServicePrincipal",
"CanDelegate": false,
"Description": null,
"ConditionVersion": null,
"Condition": null
}
]
I need to add a KEY at a start with all this json as a VALUE of that key using powershell script so it would look like this:
{
"ManagedIdentity": [
{
"DisplayName": "Title1",
"SignInName": null,
"RoleDefinitionName": "Name1",
"RoleDefinitionId": "1111",
"ObjectId": "0111",
"ObjectType": "ServicePrincipal",
"CanDelegate": false,
"Description": null,
"ConditionVersion": null,
"Condition": null
},
{
"DisplayName": "Title2",
"SignInName": null,
"RoleDefinitionName": "Name2",
"RoleDefinitionId": "1111",
"ObjectId": "02222",
"ObjectType": "ServicePrincipal",
"CanDelegate": false,
"Description": null,
"ConditionVersion": null,
"Condition": null
},
{
"DisplayName": "Title3",
"SignInName": null,
"RoleDefinitionName": "Name3",
"RoleDefinitionId": "33333",
"ObjectId": "03333",
"ObjectType": "ServicePrincipal",
"CanDelegate": false,
"Description": null,
"ConditionVersion": null,
"Condition": null
}
]
}
I should do it inside one script block, and i am not sure how to figure this out. It is a ready json file that i need to append key to.
The script that gets a file is something like this:
Get-AzRoleAssignment -ObjectId 02222 | ConvertTo-Json
to this short script using pipes i need to append a solution to append the key to the file.
Guy, long term looking at this board and learning a lot but now stuck with little issue. Im working with Linux shell script that reads json (no problem here). What Im trying to do is get value from entry that has specific Type.
By parsing a json with just jq -r '.', I get
{
"records": [
{
"id": 01,
"type": "SOA",
"name": "#",
"data": "1800",
"priority": null,
"port": null,
"ttl": 1800,
"weight": null,
"flags": null,
"tag": null
},
{
"id": 02,
"type": "A",
"name": "#",
"data": "test.com",
"priority": null,
"port": null,
"ttl": 1800,
"weight": null,
"flags": null,
"tag": null
}
],
"links": {},
"meta": {
"total": 2
}
}
Then, I use "jq -r '.records' " and get:
[
{
"id": 01,
"type": "SOA",
"name": "#",
"data": "1800",
"priority": null,
"port": null,
"ttl": 1800,
"weight": null,
"flags": null,
"tag": null
},
{
"id": 02,
"type": "A",
"name": "#",
"data": "test.com",
"priority": null,
"port": null,
"ttl": 1800,
"weight": null,
"flags": null,
"tag": null
}
]
What I need to do is get data value of the type A. Currently, we have type SOA and A, but I need to only get data from A.
I can use dummy way of "jq -r '.records[1].data'" and it gives me correct response of test.com, but I want a more dynamic way of searching for specific type (in this case "A") and then giving the data value.
Thanks guys!
Is the field called domain_records or just records?
Use select to match your criteria
jq -r '.records[] | select(.type == "A").data'
test.com
Demo
Hello internet people,
Apparently I had the same problem of many but after searching a lot even here I could not find a reasonable answer at all!
My problem:
On shell I did enter the command:
stripe invoices list --customer MY_CUSTOMER_ID_HERE
Then the following JSON appeared:
{
"object": "list",
"data": [
{
"id": "INVOICE_ID",
"object": "invoice",
"account_country": "COUNTRY",
"account_name": "MY_BUSYNESS_NAME",
"account_tax_ids": null,
"amount_due": 1000,
"amount_paid": 1000,
"amount_remaining": 0,
"application_fee_amount": null,
"attempt_count": 1,
"attempted": true,
"auto_advance": false,
"billing_reason": "subscription_create",
"charge": "CHARGE_ID",
"collection_method": "charge_automatically",
"created": SOME_UNIXTIME,
"currency": "usd",
"custom_fields": null,
"customer": "CUSTOMER_ID",
"customer_address": null,
"customer_email": "CUSTOMER_EMAIL",
"customer_name": null,
"customer_phone": "CUSTOMER_PHONE",
"customer_shipping": null,
"customer_tax_exempt": "none",
"customer_tax_ids": [
],
"default_payment_method": null,
"default_source": null,
"default_tax_rates": [
],
"description": null,
"discount": null,
"discounts": [
],
"due_date": null,
"ending_balance": 0,
"footer": null,
"hosted_invoice_url": "INVOICE_URL_GOES_HERE",
"invoice_pdf": "INVOICE_PDF_URL_GOES_HERE",
"last_finalization_error": null,
"lines": {
"object": "list",
"data": [
{
"id": "ID",
"object": "line_item",
"amount": 1000,
"currency": "usd",
"description": "PLAN_DESCRIPTION",
"discount_amounts": [
],
"discountable": true,
"discounts": [
],
"livemode": false,
"metadata": {
},
"period": {
"end": SOME_UNIXTIME,
"start": SOME_UNIXTIME
},
"plan": {
"id": "PRICE_ID_HERE",
"object": "plan",
"active": true,
"aggregate_usage": null,
"amount": 1000,
"amount_decimal": "1000",
"billing_scheme": "per_unit",
"created": SOME_UNIXTIME,
"currency": "usd",
"interval": "month",
"interval_count": 1,
"livemode": false,
"metadata": {
},
"nickname": null,
"product": "PRODUCT_ID_HERE",
"tiers_mode": null,
"transform_usage": null,
"trial_period_days": null,
"usage_type": "licensed"
},
"price": {
"id": "MORE_ID_HERE",
"object": "price",
"active": true,
"billing_scheme": "per_unit",
"created": SOME_UNIXTIME,
"currency": "usd",
"livemode": false,
"lookup_key": null,
"metadata": {
},
"nickname": null,
"product": "PRODUCT_ID_HERE",
"recurring": {
"aggregate_usage": null,
"interval": "month",
"interval_count": 1,
"trial_period_days": null,
"usage_type": "licensed"
},
"tiers_mode": null,
"transform_quantity": null,
"type": "recurring",
"unit_amount": 1000,
"unit_amount_decimal": "1000"
},
"proration": false,
"quantity": 1,
"subscription": "SUBSCRIPTION_ID_HERE",
"subscription_item": "SUBS_ITEM_ID_HERE",
"tax_amounts": [
],
"tax_rates": [
],
"type": "subscription"
}
],
"has_more": false,
"total_count": 1,
"url": "IV_URL_HERE"
},
"livemode": false,
"metadata": {
},
"next_payment_attempt": null,
"number": "NUMBER_IV_XXXX",
"on_behalf_of": null,
"paid": true,
"payment_intent": "PAYMENT_INTENT_ID_HERE",
"payment_settings": {
"payment_method_options": null,
"payment_method_types": null
},
"period_end": SOME_UNIXTIME,
"period_start": SOME_UNIXTIME,
"post_payment_credit_notes_amount": 0,
"pre_payment_credit_notes_amount": 0,
"receipt_number": null,
"starting_balance": 0,
"statement_descriptor": null,
"status": "paid",
"status_transitions": {
"finalized_at": SOME_UNIXTIME,
"marked_uncollectible_at": null,
"paid_at": SOME_UNIXTIME,
"voided_at": null
},
"subscription": "SUBSCRIPTION_ID_HERE",
"subtotal": 1000,
"tax": null,
"total": 1000,
"total_discount_amounts": [
],
"total_tax_amounts": [
],
"transfer_data": null,
"webhooks_delivered_at": SOME_UNIXTIME
}
],
"has_more": false,
"url": "/some_path"
}
I'm able to access the second level with this command:
stripe invoices list --customer MY_CUSTOMER_ID_HERE | [.id,.attempt_count,.billing_reason,.created,.customer,.customer_email,.lines.data[],.paid,.status]'
Which returns a nice response, but I do want to filter few values inside data (data.lines.data[]) like id, plan and inside plan the id, currency and product for example.
My question:
Does any one knows how to do that with this kind of commands?
Just in case, those are the questions and answer that I tried but couldn't figured how to do it at all!
How to filter array of objects by element property values using jq?
How to filter arrays in json based on specific value using jq
How to filter nested arrays in JSON while maintaining structure using jq
filtering from JSON output from curl using JQ
Filtering JSON by object name using jq
How to filter JSON using jq stream (duplicate)
How to filter and replace values in json with jq
Filtering JSON list in shell using jq
I did also try a lot attempts on my own, like:
stripe invoices list --customer MY_CUSTOMER_ID_HERE | [.id,.attempt_count,.billing_reason,.created,.customer,.customer_email,.lines.data[.id.plan[][.id,.plan[.id,.currency,.product]],.paid,.status]'
Response:
jq: error (at :179): Cannot index string with string "plan"
stripe invoices list --customer MY_CUSTOMER_ID_HERE | [.id,.attempt_count,.billing_reason,.created,.customer,.customer_email,.lines.data[][],.paid,.status]'
This one return a null value for the second data content.
Response:
[
"DESIRED_ID",
1,
"value_retrieved_ok",
unixtime_ok,
"SOME_ID_OK",
"customer_mail#ok.com",
null, <<< ???
true,
"paid"
]
EDIT:
Desired result should look like:
[
"DESIRED_ID",
1,
"value_retrieved_ok",
unixtime_ok,
"SOME_ID_OK",
"customer_mail#ok.com",
"id" // the id inside lines.data{}
["id", // the id inside lines.data.plan
"currency", // the currency inside lines.data.plan
"product" // the product inside lines.data.plan
]
true,
"paid"
]
Anyway I do hope that there's a good soul out there have mercy!
Your requirements are a bit difficult to follow, especially since the "expected output" does not correspond exactly to the given input, but the following is either what you're looking for, or very close to it:
.data[]
| [.id,.attempt_count,.billing_reason,.created,.customer,.customer_email]
+ (.lines.data[] | [.id, [.plan.id, .plan.currency, .plan.product]])
+ [.paid,.status]
I have a sourcetype defined like this (system\local\props.conf):
[my_json]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = json
NO_BINARY_CHECK = true
TIMESTAMP_FIELDS = Timestamp
category = Structured
description = json
disabled = false
pulldown_type = 1
TIME_FORMAT = HH:mm:ss.fff
LINE_BREAKER = ([\r\n]+)
limits.conf:
[spath]
# Number of characters to read from an XML or JSON event when
# auto extracting.
extraction_cutoff = 5000
extract_all = true
If I try to index following json (no line breaks, I just formated it here):
{
"Timestamp": "19:51:27.757",
"Level": "INFO",
"EventType": "Audit",
"EventId": "ApiServiceInvocationResponse",
"ThreadId": "19",
"Method": "TXXXX1234.Common.WCF.ParameterInspector.AfterCall",
"Context": {
"PhoneNumber": "48600000000",
"ApplicationId": "7C217CF0CC45E0292623203E56AD87EC",
"ApiType": "android",
"ApiVersion": "6.0",
"AppVersion": "1.0.debug",
"UserId": 25714,
"SessionId": 1440538,
"CorrelationId": "98ccaec5-4d23-4c5f-b5da-7ce0e440f2e3"
},
"Payload": {
"Operation": "Initialize",
"Response": {
"Message": null,
"CanRun": true,
"PhoneNumber": null,
"DefaultPhoneNumber": "48600000000",
"DriverPhoneNumber": null,
"RegisterationPhoneNumber": null,
"Favourites": null,
"SessionId": "4DC24EB6E4B0261DD03CDD4F6A7C7DC8",
"IsFriendlyCustomer": true,
"OptionsAvailable": [],
"MaxOrderDate": null,
"FavouriteDriverNumber": null,
"ShareMessage": null,
"PaymentInstruments": [],
"InAppPaymentAvailable": true,
"HasActiveOrders": false,
"UserName": "some name",
"UserPhone": "48600000000",
"ApplicationId": "",
"KioskInfo": null,
"CallResult": {
"Code": "SSREA",
"Message": null
}
},
"truncate": false
},
"Message": null,
"Exception": null
}
it gets properly indexed. But the following one does NOT get indexed:
{
"Timestamp": "16:31:27.074",
"Level": "INFO",
"EventType": "Audit",
"EventId": "ApiServiceInvocationResponse",
"ThreadId": "5",
"Method": "TXXXX1234.Common.WCF.ParameterInspector.AfterCall",
"Context": {
"PhoneNumber": "48600000000",
"ApplicationId": "A70BAFD855CE7120A8E331E27D39E645",
"ApiType": "MOCK",
"ApiVersion": "1.0",
"AppVersion": null,
"UserId": 11852,
"SessionId": 448107,
"CorrelationId": "28d9cc6f-c207-4199-9c24-ac6c4b4cfc8e"
},
"Payload": {
"Operation": "Initialize",
"Response": {
"Message": "message",
"CanRun": false,
"PhoneNumber": "48600000000",
"DefaultPhoneNumber": "48600000000",
"DriverPhoneNumber": "",
"RegisterationPhoneNumber": null,
"Favourites": [],
"SessionId": "0778662D04444C9456694B3FAB44F8C6",
"IsFriendlyCustomer": true,
"OptionsAvailable": [
"PaymentCard",
"Combi",
"SevenSeats",
"Animal",
"AirContition"
],
"MaxOrderDate": "2019-01-30 16:31",
"FavouriteDriverNumber": null,
"ShareMessage": "some long share message. http://www.sharing.net.pl/",
"PaymentInstruments": [],
"InAppPaymentAvailable": false,
"HasActiveOrders": false,
"UserName": "some name",
"UserPhone": "48600000000",
"ApplicationId": "",
"KioskInfo": null,
"CallResult": {
"Code": "SSREA",
"Message": "Zwr󣮯 klucz sesji dla zarejestrowanego uzytkownika"
}
},
"truncate": false
},
"Message": null,
"Exception": null
}
UPDATE:
This is what I have found in logs:
01-02-2019 20:40:31.780 +0100 ERROR JsonLineBreaker - JSON StreamId:9928927958268928125 had parsing error:Unexpected character while parsing backslash escape: 'x' - data_source="C:\Logs\Txxx.log", data_host="WIN-BP2MBISNI04", data_sourcetype="my_json"
Try adding CHARSET = UTF-8 to the props.conf stanza.
I have following JSON file and would like to extract part of it:
{
"expand": "names,schema",
"issues": [
{
"expand": "operations,versionedRepresentations,editmeta,changelog,renderedFields",
"fields": {
"aggregateprogress": {
"progress": 0,
"total": 0
},
"aggregatetimeestimate": null,
"aggregatetimeoriginalestimate": null,
"aggregatetimespent": null,
"assignee": {
"active": true,
"avatarUrls": {
"16x16": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=16",
"24x24": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=24",
"32x32": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=32",
"48x48": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=48"
},
"displayName": "user",
"emailAddress": "user#company.com",
"key": "user",
"name": "user",
"self": "https://jira.corp.company.com/rest/api/2/user?username=dvucanovic",
"timeZone": "Europe/Belgrade"
},
"components": [],
"created": "2018-03-06T21:24:41.000+0000",
"creator": {
"active": true,
"avatarUrls": {
"16x16": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=16",
"24x24": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=24",
"32x32": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=32",
"48x48": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=48"
},
"displayName": "user",
"emailAddress": "user#company.com",
"key": "user",
"name": "user",
"self": "https://jira.corp.company.com/rest/api/2/user?username=dvucanovic",
"timeZone": "Europe/Belgrade"
},
"customfield_10000": null,
"customfield_10001": null,
"customfield_10002": null,
"customfield_10004": "0|i00uta:",
"customfield_10005": null,
"customfield_10006": null,
"customfield_10100": null,
"customfield_10101": [],
"customfield_10102": null,
"customfield_10103": null,
"customfield_10107": {
"id": "10400",
"self": "https://jira.corp.company.com/rest/api/2/customFieldOption/10400",
"value": "Hentsu Internal"
},
"customfield_10108": null,
"customfield_10200": null,
"customfield_10201": "2018-03-06",
"customfield_10202": "2018-03-06",
"customfield_10203": null,
"customfield_10204": null,
"customfield_10205": null,
"customfield_10206": null,
"customfield_10300": "com.atlassian.servicedesk.plugins.approvals.internal.customfield.ApprovalsCFValue#5e6792fb",
"customfield_10301": null,
"customfield_10302": null,
"customfield_10600": null,
"customfield_10700": null,
"customfield_11000": null,
"customfield_11001": null,
"customfield_11002": null,
"customfield_11003": null,
"customfield_11004": null,
"customfield_11005": null,
"customfield_11006": null,
"customfield_11007": null,
"customfield_11008": null,
"customfield_11009": null,
"customfield_11010": null,
"customfield_11011": null,
"customfield_11012": null,
"customfield_11013": null,
"customfield_11014": null,
"customfield_11015": null,
"customfield_11016": null,
"customfield_11017": null,
"customfield_11018": null,
"customfield_11019": null,
"customfield_11100": null,
"customfield_11101": null,
"customfield_11102": null,
"description": null,
"duedate": null,
"environment": null,
"fixVersions": [],
"issuelinks": [],
"issuetype": {
"avatarId": 10318,
"description": "A task that needs to be done.",
"iconUrl": "https://jira.corp.company.com/secure/viewavatar?size=xsmall&avatarId=10318&avatarType=issuetype",
"id": "10100",
"name": "Task",
"self": "https://jira.corp.company.com/rest/api/2/issuetype/10100",
"subtask": false
},
"labels": [],
"lastViewed": "2018-03-06T21:31:34.315+0000",
"priority": {
"iconUrl": "https://jira.corp.company.com/images/icons/priorities/medium.svg",
"id": "3",
"name": "Medium",
"self": "https://jira.corp.company.com/rest/api/2/priority/3"
},
"progress": {
"progress": 0,
"total": 0
},
"project": {
"avatarUrls": {
"16x16": "https://jira.corp.company.com/secure/projectavatar?size=xsmall&pid=10001&avatarId=10201",
"24x24": "https://jira.corp.company.com/secure/projectavatar?size=small&pid=10001&avatarId=10201",
"32x32": "https://jira.corp.company.com/secure/projectavatar?size=medium&pid=10001&avatarId=10201",
"48x48": "https://jira.corp.company.com/secure/projectavatar?pid=10001&avatarId=10201"
},
"id": "10001",
"key": "TECH",
"name": "Technology",
"self": "https://jira.corp.company.com/rest/api/2/project/10001"
},
"reporter": {
"active": true,
"avatarUrls": {
"16x16": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=16",
"24x24": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=24",
"32x32": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=32",
"48x48": "https://secure.gravatar.com/avatar/5d92f3ce51d4a090cdcb9b77ee890989?d=mm&s=48"
},
"displayName": "user",
"emailAddress": "user#company.com",
"key": "user",
"name": "user",
"self": "https://jira.corp.company.com/rest/api/2/user?username=dvucanovic",
"timeZone": "Europe/Belgrade"
},
"resolution": null,
"resolutiondate": null,
"status": {
"description": "",
"iconUrl": "https://jira.corp.company.com/",
"id": "10000",
"name": "Backlog",
"self": "https://jira.corp.company.com/rest/api/2/status/10000",
"statusCategory": {
"colorName": "blue-gray",
"id": 2,
"key": "new",
"name": "To Do",
"self": "https://jira.corp.company.com/rest/api/2/statuscategory/2"
}
},
"subtasks": [
{
"fields": {
"issuetype": {
"avatarId": 10316,
"description": "The sub-task of the issue",
"iconUrl": "https://jira.corp.company.com/secure/viewavatar?size=xsmall&avatarId=10316&avatarType=issuetype",
"id": "10101",
"name": "Sub-task",
"self": "https://jira.corp.company.com/rest/api/2/issuetype/10101",
"subtask": true
},
"priority": {
"iconUrl": "https://jira.corp.company.com/images/icons/priorities/medium.svg",
"id": "3",
"name": "Medium",
"self": "https://jira.corp.company.com/rest/api/2/priority/3"
},
"status": {
"description": "",
"iconUrl": "https://jira.corp.company.com/",
"id": "10000",
"name": "Backlog",
"self": "https://jira.corp.company.com/rest/api/2/status/10000",
"statusCategory": {
"colorName": "blue-gray",
"id": 2,
"key": "new",
"name": "To Do",
"self": "https://jira.corp.company.com/rest/api/2/statuscategory/2"
}
},
"summary": "Disable user account in Local AD"
},
"id": "16916",
"key": "TECH-728",
"self": "https://jira.corp.company.com/rest/api/2/issue/16916"
}
],
"summary": "disable user mm",
"timeestimate": null,
"timeoriginalestimate": null,
"timespent": null,
"updated": "2018-03-06T21:27:22.000+0000",
"versions": [],
"votes": {
"hasVoted": false,
"self": "https://jira.corp.company.com/rest/api/2/issue/TECH-726/votes",
"votes": 0
},
"watches": {
"isWatching": true,
"self": "https://jira.corp.company.com/rest/api/2/issue/TECH-726/watchers",
"watchCount": 1
},
"workratio": -1
},
"id": "16914",
"key": "TECH-726",
"self": "https://jira.corp.company.com/rest/api/2/issue/16914"
}
],
"maxResults": 50,
"startAt": 0,
"total": 1
}
How to extract only part in subtasks section
(subtask:true) ?
tried with:
jq -r '.issues[] | .fields.issuetype.subtask' 1.json # and
jq -r '.subtasks[] |.fields.issuetype.subtask' 1.json
but always get false
jq -r '.subtasks[] | .fields.issuetype.subtask' 1.json
jq: error (at 1.json:231): Cannot iterate over null (null)
jq solution:
jq '.issues[].fields.subtasks[].fields | .issuetype.subtask, .summary' 1.json
The output:
true
"Disable user account in Local AD"
With respect to a best-practice mechanism to iterate over output in bash (assuming, here, that the subtask field will never contain a literal tab, and that summary will never contain a literal newline):
while IFS=$'\t' read -r subtask summary; do
echo "Got subtask $subtask and summary $summary"
done < <(jq -r '.issues[].fields.subtasks[].fields
| [.issuetype.subtask, .summary] | #tsv')
If you want to avoid those assumptions, you can do even better by using NUL delimiters, which will correctly handle all values which can potentially be stored in a variable in bash:
while IFS= read -r -d '' subtask && IFS= read -r -d '' summary; do
printf 'Got subtask %q and summary %q\n' "$subtask" "$summary"
done < <(jq -j '.issues[].fields.subtasks[].fields
| (.issuetype.subtask, "\u0000", .summary, "\u0000")')
Community Wiki since this answer is intended to address an issue observed in comments on the accepted answer, rather than addressing the original question as such.