How to confirm Webhook POST is being received in Coldfusion - json

To be clear upfront, I'm not a professional - just trying to get a task done and I have minimal expertise. What I thought would be easy is apparently not so. I've dealt with APIs and XML but it's always been me initiating. This is apparently different.
Goal:
Receive HTTP POST Webhook data (JSON) from my Wordpress/Woocommerce website.
Problem:
The Webhook is working and being received perfectly by RequestBin.The JSON string is being sent over HTTP. But I'm not even sure the request is being heard by my CF Template. When I try to write the received data into a session variable so I can at least see it, nothing is there?!?! How do I find out if my code being triggered?
First line of code in my CF Template:
<cfset HTTPRequestData = deserializeJSON(ToString(getHTTPRequestData().content))>
Example from RequestBin of data being sent:
{
"id": 259609,
"parent_id": 0,
"number": "CW-104-270219",
"order_key": "xxxxxxxxxxxx",
"created_via": "checkout",
"version": "3.5.5",
"status": "completed",
"currency": "USD",
"date_created": "2019-02-27T15:11:58",
"date_created_gmt": "2019-02-27T20:11:58",
"date_modified": "2019-02-27T15:11:59",
"date_modified_gmt": "2019-02-27T20:11:59",
"discount_total": "0.00",
"discount_tax": "0.00",
"shipping_total": "0.00",
"shipping_tax": "0.00",
"cart_tax": "0.00",
"total": "9.68",
"total_tax": "0.00",
"prices_include_tax": false,
"customer_id": 2,
"customer_ip_address": "xxxxxxxxxxxx",
"customer_user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36",
"customer_note": "",
"billing": {
"first_name": "Valerie",
"last_name": "Criswell",
"company": "",
"address_1": "",
"address_2": "",
"city": "",
"state": "",
"postcode": "",
"country": "",
"email": "xxxxxxxxxx",
"phone": ""
},
"shipping": {
"first_name": "",
"last_name": "",
"company": "",
"address_1": "",
"address_2": "",
"city": "",
"state": "",
"postcode": "",
"country": ""
},
"payment_method": "authorize_net_aim",
"payment_method_title": "Credit Card",
"transaction_id": "xxxxxxxxxxxxx",
"date_paid": "2019-02-27T15:11:59",
"date_paid_gmt": "2019-02-27T20:11:59",
"date_completed": "2019-02-27T15:11:59",
"date_completed_gmt": "2019-02-27T20:11:59",
"cart_hash": "xxxxxxxxxxxxx",
"meta_data": [
{
"id": 8785641,
"key": "_wcson_order_number",
"value": "CWD-259609-022719-108"
},
{
"id": 8785642,
"key": "_order_number",
"value": "104"
},
{
"id": 8785643,
"key": "_order_number_formatted",
"value": "CW-104-270219"
},
{
"id": 8785644,
"key": "_order_number_meta",
"value": {
"prefix": "CW-",
"suffix": "-{DD}{MM}{YY}",
"length": "2"
}
},
{
"id": 8785645,
"key": "_wc_authorize_net_aim_retry_count",
"value": "0"
},
{
"id": 8785646,
"key": "_wc_authorize_net_aim_trans_id",
"value": "xxxxxxxx"
},
{
"id": 8785647,
"key": "_wc_authorize_net_aim_trans_date",
"value": "2019-02-27 15:11:59"
},
{
"id": 8785648,
"key": "_wc_authorize_net_aim_environment",
"value": "production"
},
{
"id": 8785649,
"key": "_wc_authorize_net_aim_account_four",
"value": "xxxxxxxx"
},
{
"id": 8785650,
"key": "_wc_authorize_net_aim_authorization_amount",
"value": "xxxxxxx"
},
{
"id": 8785651,
"key": "_wc_authorize_net_aim_authorization_code",
"value": "xxxxxxx"
},
{
"id": 8785652,
"key": "_wc_authorize_net_aim_charge_captured",
"value": "yes"
},
{
"id": 8785653,
"key": "_wc_authorize_net_aim_card_expiry_date",
"value": "21-08"
},
{
"id": 8785654,
"key": "_wc_authorize_net_aim_card_type",
"value": "mastercard"
},
{
"id": 8785659,
"key": "_pip_invoice_number",
"value": "CW-104-270219"
},
{
"id": 8785660,
"key": "_wc_pip_invoice_email_count",
"value": "1"
},
{
"id": 8785661,
"key": "_wc_pip_packing_list_email_count",
"value": "1"
}
],
"line_items": [
{
"id": 17,
"name": "I Give Myself Away - MP3 Download - 30 Day Access",
"product_id": 192127,
"variation_id": 192128,
"quantity": 1,
"tax_class": "",
"subtotal": "9.68",
"subtotal_tax": "0.00",
"total": "9.68",
"total_tax": "0.00",
"taxes": [],
"meta_data": [
{
"id": 182,
"key": "pa_format",
"value": "download-30-day"
}
],
"sku": "131599-DL30DAY",
"price": 9.68
}
],
"tax_lines": [],
"shipping_lines": [],
"fee_lines": [],
"coupon_lines": [],
"refunds": []
}

Let's start with the stated goal:
Receive HTTP POST Webhook data from my Wordpress/Woocommerce website.
Can your Wordpress site hit the intended URL on your ColdFusion site? If so, can you record / dump the HTTP POST request to a file?
<cfdump var="#form#" format="html" output="/full/path/to/file">
This will tell you if you're even receiving the request correctly.

Related

Parsing JSON with VBA-Web and VBA JSON Parser

Need help on parsing JSON data using VBA-Web tool (https://github.com/VBA-tools/VBA-Web) & VBA-JSON Converter
Here is my sample code that I am able to pull the order using the get method but I stumped on parsing the JSON to get the key, value pair that I want
Sub TestWC()
Dim WoocommerceClient As New WebClient
Dim Client As String
Dim Secret As String
WoocommerceClient.BaseUrl = "https://www.example.com"
Client = "ck"
Secret = "cs"
Dim Request As New WebRequest
Request.Method = Httpget
Request.AddQuerystringParam "consumer_key", Client
Request.AddQuerystringParam "consumer_secret", Secret
Dim Response As WebResponse
Request.Resource = "/wp-json/wc/v3/orders"
Set Response = WoocommerceClient.Execute(Request)
//Response.Data -> Should be equal to the JSON data
Dim test As Object
Set test = JsonConverter.ParseJson(Response.Data)
End Sub
When I run this code it will have a error popup saying object required 424 . I also did another work around where I put into the first cell the whole JSON Data using Sheet1.Range("A1").Value = Response.Content then by pulling the data from the cell into the function
Set test = JsonConverter.ParseJson(Sheet1.Range("A1").Value)
I still get the object required error, here is a Sample JSON format from Woocommerce REST API
"id": 727,
"parent_id": 0,
"number": "727",
"order_key": "wc_order_58d2d042d1d",
"created_via": "rest-api",
"version": "3.0.0",
"status": "processing",
"currency": "USD",
"date_created": "2017-03-22T16:28:02",
"date_created_gmt": "2017-03-22T19:28:02",
"date_modified": "2017-03-22T16:28:08",
"date_modified_gmt": "2017-03-22T19:28:08",
"discount_total": "0.00",
"discount_tax": "0.00",
"shipping_total": "10.00",
"shipping_tax": "0.00",
"cart_tax": "1.35",
"total": "29.35",
"total_tax": "1.35",
"prices_include_tax": false,
"customer_id": 0,
"customer_ip_address": "",
"customer_user_agent": "",
"customer_note": "",
"billing": {
"first_name": "John",
"last_name": "Doe",
"company": "",
"address_1": "969 Market",
"address_2": "",
"city": "San Francisco",
"state": "CA",
"postcode": "94103",
"country": "US",
"email": "john.doe#example.com",
"phone": "(555) 555-5555"
},
"shipping": {
"first_name": "John",
"last_name": "Doe",
"company": "",
"address_1": "969 Market",
"address_2": "",
"city": "San Francisco",
"state": "CA",
"postcode": "94103",
"country": "US"
},
"payment_method": "bacs",
"payment_method_title": "Direct Bank Transfer",
"transaction_id": "",
"date_paid": "2017-03-22T16:28:08",
"date_paid_gmt": "2017-03-22T19:28:08",
"date_completed": null,
"date_completed_gmt": null,
"cart_hash": "",
"meta_data": [
{
"id": 13106,
"key": "_download_permissions_granted",
"value": "yes"
}
],
"line_items": [
{
"id": 315,
"name": "Woo Single #1",
"product_id": 93,
"variation_id": 0,
"quantity": 2,
"tax_class": "",
"subtotal": "6.00",
"subtotal_tax": "0.45",
"total": "6.00",
"total_tax": "0.45",
"taxes": [
{
"id": 75,
"total": "0.45",
"subtotal": "0.45"
}
],
"meta_data": [],
"sku": "",
"price": 3
},
{
"id": 316,
"name": "Ship Your Idea – Color: Black, Size: M Test",
"product_id": 22,
"variation_id": 23,
"quantity": 1,
"tax_class": "",
"subtotal": "12.00",
"subtotal_tax": "0.90",
"total": "12.00",
"total_tax": "0.90",
"taxes": [
{
"id": 75,
"total": "0.9",
"subtotal": "0.9"
}
],
"meta_data": [
{
"id": 2095,
"key": "pa_color",
"value": "black"
},
{
"id": 2096,
"key": "size",
"value": "M Test"
}
],
"sku": "Bar3",
"price": 12
}
],
"tax_lines": [
{
"id": 318,
"rate_code": "US-CA-STATE TAX",
"rate_id": 75,
"label": "State Tax",
"compound": false,
"tax_total": "1.35",
"shipping_tax_total": "0.00",
"meta_data": []
}
],
"shipping_lines": [
{
"id": 317,
"method_title": "Flat Rate",
"method_id": "flat_rate",
"total": "10.00",
"total_tax": "0.00",
"taxes": [],
"meta_data": []
}
],
"fee_lines": [],
"coupon_lines": [],
"refunds": [],
"_links": {
"self": [
{
"href": "https://example.com/wp-json/wc/v3/orders/727"
}
],
"collection": [
{
"href": "https://example.com/wp-json/wc/v3/orders"
}
]
}
}```

Error parsing a specific JSON file in Snowflake with File Format

I have created a Stage and File Format in Snowflake which works with all my other JSON files except this, which throws an error:
Error parsing JSON: misplaced { File 'rooms.json.gz', line 1,
character 2 Row 0, column $1
I am using the same query that I am using for other files.
SELECT $1
FROM #MySchema.MY_STAGE/rooms.json.gz
;
What is wrong with the structure of this specific JSON file?
{
"rooms": [
{
"area": 131.49,
"longDescription": "",
"dateCreated": 1589908063390,
"reservable": false,
"name": "E249",
"remoteInfo": "",
"description": "",
"id": 2,
"type": {
"hexColor": "c16058",
"contentFlag": 1,
"cost": 0.0,
"dateCreated": 1308610520717,
"color": {},
"name": "BREAK ROOM",
"occupiable": false,
"id": 120,
"parkingSpace": false,
"dateUpdated": 1591818585913,
"typeCode": ""
},
"floor": {
"area": 25312.9878,
"dateCreated": 1589907703870,
"drawingAvailable": true,
"interiorGross": 0.0,
"name": "2",
"leaseArea": 0.0,
"id": 12,
"building": {
"address": {
"country": {
"defaultSelected": true,
"subdivisionCategoryName": "state",
"alpha2Code": "US",
"isoCode": "US",
"name": "United States of America (the)",
"id": 223
},
"city": "Some City",
"street": "Some Drive",
"postalCode": "00000",
"state": {
"country": {
"defaultSelected": true,
"subdivisionCategoryName": "state",
"alpha2Code": "US",
"isoCode": "US",
"name": "United States of America (the)",
"id": 223
},
"defaultSelected": false,
"code": "XX",
"name": "Some State",
"id": 66,
"categoryName": "state"
}
},
"code": "B2",
"dateCreated": 1589907508020,
"metric": false,
"name": "Some name",
"location": {},
"revitLink": "",
"id": 45,
"dateUpdated": 1601315841453,
"costCenters": []
},
"dateUpdated": 1600441936663
},
"capacity": 0,
"dateUpdated": 1600441936960
}
]
}
Edit: Screenshot from Notepad++ with all characters enabled

Custom JSON format webhook WooCommerce Wordpress

I want my Woocommerce store connected to a shipping parcel company. There is no existing WooCommerce plugin for it. I want to do it with webhooks, I had already successfully made a POST request to a URL after the order.created action. The JSON output contains all the shipping information. But I want to change the format of the JSON output.
The current JSON output is something like this.
{
"id": 605,
"parent_id": 0,
"status": "pending",
"order_key": "wc_order_5893614a8fb78",
"number": 605,
"currency": "USD",
"version": "2.6.13",
"prices_include_tax": false,
"date_created": "2017-02-02T16:41:46",
"date_modified": "2017-02-02T16:41:46",
"customer_id": 2,
"discount_total": "0.00",
"discount_tax": "0.00",
"shipping_total": "13.60",
"shipping_tax": "1.22",
"cart_tax": "1.44",
"total": "32.24",
"total_tax": "2.66",
"billing": {
"first_name": "Damandeep",
"last_name": "Singh",
"company": "",
"address_1": "1257 ",
"address_2": "Coach House Court",
"city": "Fullerton",
"state": "CA",
"postcode": "92831",
"country": "US",
"email": "daman#singhdd.com",
"phone": "(122) 274-5555"
},
"shipping": {
"first_name": "Damandeep",
"last_name": "Singh",
"company": "",
"address_1": "1257",
"address_2": "Coach House Court",
"city": "Fullerton",
"state": "CA",
"postcode": "92831",
"country": "US"
},
"payment_method": "paypal_express",
"payment_method_title": "Paypal Express",
"transaction_id": "",
"customer_ip_address": "103.41.36.35",
"customer_user_agent": "PostmanRuntime/3.0.9",
"created_via": "rest-api",
"customer_note": "",
"date_completed": "2017-02-02T08:41:46",
"date_paid": "",
"cart_hash": "",
"line_items": [
{
"id": 79,
"name": "Kaju Katli",
"sku": "SW-282",
"product_id": 491,
"variation_id": 494,
"quantity": 1,
"tax_class": "",
"price": "15.98",
"subtotal": "15.98",
"subtotal_tax": "1.44",
"total": "15.98",
"total_tax": "1.44",
"taxes": [
{
"id": 1,
"total": 1.4382,
"subtotal": 1.4382
}
],
"meta": [
{
"key": "packing-size",
"label": "Packing Size",
"value": "2lb Box"
}
]
}
],
"tax_lines": [
{
"id": 81,
"rate_code": "SALES TAX-1",
"rate_id": "1",
"label": "Sales Tax",
"compound": false,
"tax_total": "1.44",
"shipping_tax_total": "1.22"
}
],
"shipping_lines": [
{
"id": 80,
"method_title": "USPS Medium Flat Rate Box",
"method_id": "usps_medium_box",
"total": "13.60",
"total_tax": "0.00",
"taxes": []
}
],
"fee_lines": [],
"coupon_lines": [],
"refunds": [],
"_links": {
"self": [
{
"href": "https://ambala.webdemos.cf/wp-json/wc/v1/orders/605"
}
],
"collection": [
{
"href": "https://ambala.webdemos.cf/wp-json/wc/v1/orders"
}
],
"customer": [
{
"href": "https://ambala.webdemos.cf/wp-json/wc/v1/customers/2"
}
]
}
}
The JSON output should be something like this.
{
"name": "Gijs Boersma",
"street": "Lange laan",
"house_number": "29",
"house_number_extension": "a",
"zipcode": "9281EM",
"city": "Zevenaar",
"telephone": "0602938172",
"email": "noreply#example.com",
"reference": "Bestelling 112",
"pick_up_point": {
"uuid": "560db083-d941-425b-b3b6-b813718297e1"
},
"product": "sameday_parcel_medium",
"product_options": [
{
"option": "allow_neighbours",
"value": false
},
{
"option": "require_signature",
"value": false
},
{
"option": "age_check_18",
"value": false
},
{
"option": "perishable",
"value": true,
"max_attempts": 2
}
]
}
you can do this by the orders filter.
add_filter( 'woocommerce_rest_prepare_shop_order_object', 'change_shop_order_response', 10, 3 );
function change_shop_order_response( $response, $item, $request ) {
//Do your stuff here with the $response object.
return $response
}

BigQuery Error JSON_EXTRACT although jsonPath is valid

I'm trying to extract some information out of the httparchive har tables, which are provided for free.
Trouble is that BQ returns an error on a JSON_EXTRACT() function although a number of jsonPath validators are executing it perfectly (http://jsonpath.com/ or jsonpath.curiousconcept.com/).
This is my query:
SELECT
JSON_EXTRACT(payload,"$._host")AS host,
JSON_EXTRACT(payload,"$.request.headers[?(#.name=='Referer')]")AS referer,
url,
payload
FROM
[httparchive:har.2016_01_01_chrome_requests]
LIMIT
100
expected return based on the example payload json below would be
"http://www.echosdunet.net/"
This is the error:
Error: JSONPath parse error at: [?(#.name=='Referer')]
This is an example payload json:
{
"pageref": "page_1_0",
"startedDateTime": "2016-01-03T22:18:52.632+00:00",
"time": 452,
"request": {
"method": "GET",
"url": "http://disqus.com/embed/comments/?base=default&version=f3e1717b71e7256da258d3a504e56865&f=echosdunet&t_i=node%2F19849&t_u=http%3A%2F%2Fwww.echosdunet.net%2Fnode%2F19849&t_e=Accueil&t_d=Comparatif%20et%20test%20ADSL%20et%20fibre&t_t=Accueil&s_o=default&l=fr",
"headersSize": 650,
"bodySize": -1,
"cookies": [],
"headers": [{
"name": "Host",
"value": "disqus.com"
}, {
"name": "Connection",
"value": "keep-alive"
}, {
"name": "Accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
}, {
"name": "Upgrade-Insecure-Requests",
"value": "1"
}, {
"name": "User-Agent",
"value": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36 PTST/254"
}, {
"name": "Referer",
"value": "http://www.echosdunet.net/"
}, {
"name": "Accept-Encoding",
"value": "gzip, deflate, sdch"
}, {
"name": "Accept-Language",
"value": "en-US,en;q=0.8"
}],
"httpVersion": "1.1",
"queryString": [{
"name": "base",
"value": "default"
}, {
"name": "version",
"value": "f3e1717b71e7256da258d3a504e56865"
}, {
"name": "f",
"value": "echosdunet"
}, {
"name": "t_i",
"value": "node/19849"
}, {
"name": "t_u",
"value": "http://www.echosdunet.net/node/19849"
}, {
"name": "t_e",
"value": "Accueil"
}, {
"name": "t_d",
"value": "Comparatif et test ADSL et fibre"
}, {
"name": "t_t",
"value": "Accueil"
}, {
"name": "s_o",
"value": "default"
}, {
"name": "l",
"value": "fr"
}]
},
"response": {
"status": 200,
"statusText": "",
"headersSize": 1161,
"bodySize": 2017,
"headers": [{
"name": "Server",
"value": "nginx"
}, {
"name": "Content-Type",
"value": "text/html; charset=utf-8"
}, {
"name": "Content-Security-Policy",
"value": "script-src https://*.twitter.com:* https://api.adsnative.com/v1/ad.json *.adsafeprotected.com https://cas.criteo.com/delivery/0.1/napi.jsonp *.services.disqus.com:* http://referrer.disqus.com/juggler/ disqus.com http://*.twitter.com:* a.disquscdn.com https://referrer.disqus.com/juggler/ https://*.services.disqus.com:* *.moatads.com 'unsafe-eval' https://mobile.adnxs.com/mob https://ssl.google-analytics.com"
}, {
"name": "Link",
"value": "<http://a.disquscdn.com>;rel=preconnect,<http://a.disquscdn.com>;rel=dns-prefetch"
}, {
"name": "Cache-Control",
"value": "stale-if-error=3600, s-stalewhilerevalidate=3600, stale-while-revalidate=30, no-cache, must-revalidate, public, s-maxage=5"
}, {
"name": "p3p",
"value": "CP=\\DSP IDC CUR ADM DELi STP NAV COM UNI INT PHY DEM\\"
}, {
"name": "Timing-Allow-Origin",
"value": "*"
}, {
"name": "X-Content-Type-Options",
"value": "nosniff"
}, {
"name": "X-XSS-Protection",
"value": "1; mode=block"
}, {
"name": "Last-Modified",
"value": "Thu, 11 Jun 2015 13:30:36 GMT"
}, {
"name": "ETag",
"value": "W/\\lounge:view:3840102421.d93d9c4bc037078ffc811833ae267a6f.0\\"
}, {
"name": "Content-Encoding",
"value": "gzip"
}, {
"name": "Content-Length",
"value": "2017"
}, {
"name": "Accept-Ranges",
"value": "bytes"
}, {
"name": "Date",
"value": "Sun, 03 Jan 2016 22:18:51 GMT"
}, {
"name": "Age",
"value": "0"
}, {
"name": "Connection",
"value": "keep-alive"
}, {
"name": "Vary",
"value": "Accept-Encoding"
}],
"httpVersion": "1.1",
"redirectURL": "",
"content": {
"size": 2017,
"mimeType": "text/html"
},
"cookies": []
},
"cache": {},
"timings": {
"blocked": -1,
"dns": 202,
"connect": 32,
"ssl": -1,
"send": 0,
"wait": 108,
"receive": 110
},
"_ip_addr": "104.156.81.134",
"_method": "GET",
"_host": "disqus.com",
"_url": "/embed/comments/?base=default&version=f3e1717b71e7256da258d3a504e56865&f=echosdunet&t_i=node%2F19849&t_u=http%3A%2F%2Fwww.echosdunet.net%2Fnode%2F19849&t_e=Accueil&t_d=Comparatif%20et%20test%20ADSL%20et%20fibre&t_t=Accueil&s_o=default&l=fr",
"_responseCode": "200",
"_load_ms": "218",
"_ttfb_ms": "108",
"_load_start": "2632",
"_bytesOut": "652",
"_bytesIn": "3180",
"_objectSize": "2017",
"_cacheControl": "stale-if-error=3600, s-stalewhilerevalidate=3600, stale-while-revalidate=30, no-cache, must-revalidate, public, s-maxage=5",
"_contentType": "text/html",
"_contentEncoding": "gzip",
"_type": "3",
"_socket": "153",
"_score_cache": "-1",
"_score_cdn": "-1",
"_score_gzip": "100",
"_score_cookies": "-1",
"_score_keep-alive": "100",
"_score_minify": "-1",
"_score_combine": "-1",
"_score_compress": "-1",
"_score_etags": "-1",
"_is_secure": "0",
"_dns_ms": 202,
"_connect_ms": 32,
"_ssl_ms": "-1",
"_gzip_total": "3180",
"_gzip_save": "0",
"_minify_total": "0",
"_minify_save": "0",
"_image_total": "0",
"_image_save": "0",
"_cache_time": "-1",
"_dns_start": "2395",
"_dns_end": "2597",
"_connect_start": "2600",
"_connect_end": "2632",
"_ssl_start": "0",
"_ssl_end": "0",
"_initiator": "http://echosdunet.disqus.com/embed.js?_=1451859532217",
"_initiator_line": "16",
"_initiator_column": "8205",
"_server_count": "4",
"_server_rtt": "32",
"_client_port": "62284",
"_jpeg_scan_count": "0",
"_full_url": "http://disqus.com/embed/comments/?base=default&version=f3e1717b71e7256da258d3a504e56865&f=echosdunet&t_i=node%2F19849&t_u=http%3A%2F%2Fwww.echosdunet.net%2Fnode%2F19849&t_e=Accueil&t_d=Comparatif%20et%20test%20ADSL%20et%20fibre&t_t=Accueil&s_o=default&l=fr",
"_score_progressive_jpeg": -1,
"_body": true,
"_load_end": 2850,
"_ttfb_start": "2632",
"_ttfb_end": 2740,
"_download_start": 2740,
"_download_end": 2850,
"_download_ms": 110,
"_all_start": "2395",
"_all_end": 2850,
"_all_ms": 452,
"_index": 55,
"_number": 56,
"_body_url": "/response_body.php?test=160101_10_KZQ1&run=1&cached=0&request=56"
}
Running the query without the referer jsonPath works fine:
SELECT
JSON_EXTRACT(payload,"$._host")AS host,
url,
payload
FROM
[httparchive:har.2016_01_01_chrome_requests]
LIMIT
100
Meantime, try below (for BigQuery Standard SQL)
#standardSQL
CREATE TEMPORARY FUNCTION CUSTOM_JSON_EXTRACT(json STRING, key String)
RETURNS string
LANGUAGE js AS """
try {
var parsed = JSON.parse(json);
for (i = 0; i < parsed.length; i++) {
if (parsed[i].name == key) return parsed[i].value
}
} catch (e) {}
return null;
""";
SELECT
JSON_EXTRACT(payload,"$._host")AS host,
CUSTOM_JSON_EXTRACT(JSON_EXTRACT(payload, "$.request.headers"), 'Referer') AS referer,
url,
payload
FROM
`httparchive.har.2016_01_01_chrome_requests`
LIMIT
100
JSON_EXTRACT and JSON_EXTRACT_SCALAR do not support the complete set of JSONPath elements. You can read more in the documentation. I would suggest using standard SQL since that implementation of the JSON functions has better error messages.
If you are interested in additional kinds of JSONPath elements, you can consider submitting a feature request.

Convert json to include name and children values to feed D3

I am trying to get json converted from:
{
"Devices": [
{
"Udid": "7a2b0e6c928f2321a75e423ba23ae93d",
"SerialNumber": "RF1D232ZLEE",
"MacAddress": "40F232726FC8",
"Imei": "3576342323280150",
"EasId": "SEC1BC252327E92B",
"AssetNumber": "7a2b0e23223928f2321a75e423ba23ae93d",
"DeviceFriendlyName": "gel1 Android Android 5.0.1 ZLEE ",
"LocationGroupId": {
"Id": {
"Value": 19529
},
"Name": "Group Express"
},
"LocationGroupName": "Group Express",
"UserId": {
"Name": ""
},
"UserName": "",
"UserEmailAddress": "",
"Ownership": "S",
"PlatformId": {
"Id": {
"Value": 5
},
"Name": "Android"
},
"Platform": "Android",
"ModelId": {
"Id": {
"Value": 5
},
"Name": "samsung GT-I9505"
},
"Model": "samsung GT-I9505",
"OperatingSystem": "5.0.1",
"PhoneNumber": "+447881867010",
"LastSeen": "2016-07-06T14:01:03.590",
"EnrollmentStatus": "Unenrolled",
"ComplianceStatus": "NotAvailable",
"CompromisedStatus": false,
"LastEnrolledOn": "2016-06-15T16:01:38.763",
"LastComplianceCheckOn": "0001-01-01T00:00:00.000",
"LastCompromisedCheckOn": "2016-07-06T13:58:26.183",
"IsSupervised": false,
"DeviceMCC": {
"SIMMCC": "234",
"CurrentMCC": "234"
},
"AcLineStatus": 0,
"VirtualMemory": 0,
"Id": {
"Value": 23459
}
},
{
"Udid": "c5f94db71d406dae7f881d3edf059e",
"SerialNumber": "",
"MacAddress": "000C300F9108",
"Imei": "",
"EasId": "D80DB85EC411C8E9B28BC292A603F05C2C0EEEC8",
"AssetNumber": "c592f93db71d406dae7f881d3edf059e",
"DeviceFriendlyName": "user Windows 10 WinRT 10.0.10240 ",
"LocationGroupId": {
"Id": {
"Value": 18498
},
"Name": "Business Solutions"
},
"LocationGroupName": "Business Solutions",
"UserId": {
"Name": ""
},
"UserName": "",
"UserEmailAddress": "",
"Ownership": "C",
"PlatformId": {
"Id": {
"Value": 12
},
"Name": "WinRT"
},
"Platform": "WinRT",
"ModelId": {
"Id": {
"Value": 50
},
"Name": "Windows 10"
},
"Model": "Windows 10",
"OperatingSystem": "10.0.10240",
"PhoneNumber": "",
"LastSeen": "2016-05-03T10:54:07.650",
"EnrollmentStatus": "Unenrolled",
"ComplianceStatus": "NotAvailable",
"CompromisedStatus": false,
"LastEnrolledOn": "2016-01-29T16:41:57.760",
"LastComplianceCheckOn": "0001-01-01T00:00:00.000",
"LastCompromisedCheckOn": "0001-01-01T00:00:00.000",
"IsSupervised": false,
"DeviceMCC": {
"SIMMCC": "",
"CurrentMCC": ""
},
"AcLineStatus": 0,
"VirtualMemory": 0,
"Id": {
"Value": 23545
}
}
],
"Page": 0,
"PageSize": 500,
"Total": 13}
To something like:
{"name": "Devices",
"children": [
{"name":"Udid", "size":"7f0dsda63274692ea4f0b66fec67a020158"},
{"name":"SerialNumber", "size":"P988KJSPQF938"},
{"name":"MacAddress", "size":"1HJUSUD031C4"},
{"name":"Imei", "size":""},
{"name":"EasId", "size":"ApKJSPQF193"},
{"name":"AssetNumber", "size":"7f0cda636b3305fea4f0b66fec9997267a020158"},
{"name":"DeviceFriendlyName", "size":"TMcKenz iPad iOS 7.1.4 F193 "},
{"name":"LocationGroupId",
"children": [
{"name":"Id","size":7488},
{"name":"Name","size":"MCM"}
]
},
{"name":"UserId",
"children": [
{"name":"Id","size":6418},
{"name":"Name","size":"Tom McKenz"}
]
},
{"name":"UserName", "size":"TMcKenz"},
{"name":"UserEmailAddress", "size":"TMcKenz#awaw.com"}
]
}
Not sure what is the best practice here: is it possible to use D3.nest or do you need to iterate through all the nodes and change to 'name' and 'children' accordingly.