Azure API Management latency of over 5 seconds - azure-api-management

We're using API Management to execute an always on function app that, in turn, is calling Azure Search. The function app responds within 100 ms (usually well under), but API Management intermittently takes 3 to 8 seconds to process. I've gone through the trace and found the delay is consistently in the backend step in which API Management is processing the results (sample trace for backend below). The returned data in this case is 664 bytes. Does anyone know what may cause this and, more importantly, how to remove this latency?
The Azure API Management instance, function app, and Azure Search are all in the same region.
"backend": [
{
"source": "forward-request",
"timestamp": "2018-10-11T13:33:54.0578250Z",
"elapsed": "00:00:00.0005890",
"data": {
"message": "Request is being forwarded to the backend service. Timeout set to 300 seconds",
"request": {
"method": "GET",
"url": "https://ommitted.azurewebsites.net/api/HollanderSearch/?hollander=125-222L&websiteId=1&retrieveAllProductNames=false&onlyRetrieveProductNames=true&numRecsToRetrieve =25&code=%2Fxu1iz0EimDFnOUvhDVaM4ZejfaW7umV3somaIDCeWJYI5RMIfNnNg%3D%3D&keyword=125-222L",
"headers": [
{
"name": "Ocp-Apim-Subscription-Key",
"value": "ommitted"
},
{
"name": "Postman-Token",
"value": "9e809f11-6e63-4422-8c3f-d61962cec6ac"
},
{
"name": "Cache-Control",
"value": "no-cache"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate"
},
{
"name": "User-Agent",
"value": "PostmanRuntime/7.3.0"
},
{
"name": "X-Forwarded-For",
"value": "omitted"
}
]
}
}
},
{
"source": "forward-request",
"timestamp": "2018-10-11T13:33:59.6717259Z",
"elapsed": "00:00:05.6045857",
"data": {
"response": {
"status": {
"code": 200,
"reason": "OK"
},
"headers": [
{
"name": "Pragma",
"value": "no-cache"
},
{
"name": "Transfer-Encoding",
"value": "chunked"
},
{
"name": "Content-Encoding",
"value": "gzip"
},
{
"name": "Vary",
"value": "Accept-Encoding"
},
{
"name": "Cache-Control",
"value": "no-cache"
},
{
"name": "Content-Type",
"value": "application/json; charset=utf-8"
},
{
"name": "Date",
"value": "Thu, 11 Oct 2018 13:33:59 GMT"
},
{
"name": "Expires",
"value": "-1"
},
{
"name": "Set-Cookie",
"value": "ARRAffinity=8065e5ea6c0774f598f30cd38416a122cd6afe1af34a15c7f6430e1a0fe9924c;Path=/;HttpOnly;Domain=omitted-dev.azurewebsites.net"
},
{
"name": "Server",
"value": "Microsoft-IIS/10.0"
},
{
"name": "X-Powered-By",
"value": "ASP.NET"
}
]
}
}
},
{
"source": "decompressor",
"timestamp": "2018-10-11T13:33:59.6717259Z",
"elapsed": "00:00:05.6045976",
"data": "Content-Encoding header with value `gzip` detected in message headers. Content re-compression may require to correctly apply content modification policies."
}
]

Related

Power Automate Adaptive card does not display in my email

Hi I have been working with power automate to create an approval flow that is triggered by a power apps app. The email contains the json for an adaptive card used for storing responses. When I receive the email I cannot see the adaptive card. I have created a profile on the action card dashboard and it has been more than 12 hours since I was approved. I have no idea what is going on.
The below is the JSON for my Adaptive card.
{ "type": "AdaptiveCard", "$schema": "http://adaptivecards.io/schemas/adaptive-card.json", "version": "1.4", "body": [
{
"type": "TextBlock",
"text": ⁠ Compose_Inputs × ⁠,
"wrap": true,
"id": "lblID"
},
{
"type": "TextBlock",
"text": ⁠ Condition3_Value × ⁠,
"wrap": true,
"id": "lblGid"
},
{
"type": "Input.ChoiceSet",
"choices": [
{
"title": "Approve",
"value": "1"
},
{
"title": "Reject",
"value": "2"
}
],
"placeholder": "Choose whether to accept or reject the declaration",
"id": "cmbApproval"
},
{
"type": "RichTextBlock",
"inlines": [
{
"type": "TextRun",
"text": "Enter the reason for a rejection here"
}
],
"id": "redtRejection"
},
{
"type": "RichTextBlock",
"inlines": [
{
"type": "TextRun",
"text": "Reccomend some actions that need to be taken"
}
],
"id": "redtActionable"
} ], "actions": [
{
"type": "Action.Http",
"title": "Submit",
"id": "btnSubmit",
"method": "POST",
"body": "{\n\"Approval\" : \"{{cmbApproval.value}}\",\n\"Actions\": \"{{redtActionable.text}}\",\n\"Rejection\": \"{{redtRejection.text}}\",\n\"Approval_ID\":\"{{lblID.text}}\",\n\"Gift_ID\":\"{{lblGid.text}}\"\n}",
"url": "https://prod-78.westeurope.logic.azure.com:443/workflows/2ecc98c6673e4d88b2569fb6be6eae33/triggers/manual/paths/invoke?api-version=2016-06-01&sp=%2Ftriggers%2Fmanual%2Frun&sv=1.0&sig=to-6fjShHwNxMqXywd-HNg5Wl-z95eiorM0w1ZyiZTs",
"headers": [
{
"name": "Authorization",
"value": ""
},
{
"name": "Content-type",
"value": "application/json"
}
],
"originator": "1e643d3f-b41b-4723-b72d-23a5e813a5ba"
} ], "id": "btnSubmit" }

Update a Contentful post using PUT request via Postman

I'm trying to update a Contentful entry using Postman.
What I did:
In Contentful space, I created a test post to play with.
Went to Settings - API Keys - Content management tokens and generated a Personal access token
Created a GET request in Postman, passing space ID, master environment, and ID of the test post:
https://cdn.contentful.com/spaces/{spaceID}i/environments/master/entries?sys.id={postID}
I also sent authorisation header with content delivery token.
The GET request goes successfully and I'm able to copy the JSON object-response.
{
"sys": {
"type": "Array"
},
"total": 1,
"skip": 0,
"limit": 100,
"items": [
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2Fwow39hxxx1bvMkjpsyV9",
"type": "Entry",
"createdAt": "2021-11-10T14:00:11.935Z",
"updatedAt": "2021-11-10T14:06:51.393Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"revision": 3,
"contentType": {
"sys": {
"type": "Link",
"linkType": "ContentType",
"id": "hotelInfo"
}
},
"locale": "en-US"
},
"fields": {
"name": "Test entry",
"slug": "test-entry",
"address": "Lviv",
"cityName": "Lviv",
"phone": "+380931231212",
"coordinates": {
"lon": -115.302,
"lat": 36.18709
},
"dog": "100",
"cat": "100",
"delivery": "100",
"photo": [
{
"sys": {
"type": "Link",
"linkType": "Asset",
"id": "2hSnYhQDJzU99NvlsYdk3k"
}
}
],
"additionalInfo": {
"data": {},
"content": [
{
"data": {},
"content": [
{
"data": {},
"marks": [],
"value": "Test",
"nodeType": "text"
}
],
"nodeType": "paragraph"
}
],
"nodeType": "document"
},
"featuredHotel": true,
"phoneClicks": 1
}
}
],
"includes": {
"Asset": [
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2hSnYhQDJzU99NvlsYdk3k",
"type": "Asset",
"createdAt": "2021-11-10T13:59:59.954Z",
"updatedAt": "2021-11-10T13:59:59.954Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"revision": 1,
"locale": "en-US"
},
"fields": {
"title": "JS",
"description": "Lorem Ipsum",
"file": {
"url": "//images.ctfassets.net/d9r4mg123x4v/2hSnYhQDJzU99NvlsYdk3k/6fbabc7be7f4b28dc8b7deadd9892205/JS.png",
"details": {
"size": 23078,
"image": {
"width": 1024,
"height": 1024
}
},
"fileName": "JS.png",
"contentType": "image/png"
}
}
}
]
}
}
Now I want to create PUT request to send the updated JSON to the Contentful.
I paste the JSON I got as a response from GET request.
I change one of the values:
"name": "Test entry"
to
"name": "Test entry 123"
I send PUT request to https://api.contentful.com/spaces/{spaceID}/environments/master/entries/{postID}
The Authorisation header contains the Personal access token I generated before.
The X-Contentful-Version header contains the version of the post, can be found in post details
When I send this request, I get JSON response with an empty "fields": {}
{
"metadata": {
"tags": []
},
"sys": {
"space": {
"sys": {
"type": "Link",
"linkType": "Space",
"id": "d9r4mg123x4v"
}
},
"id": "2Fwow39hxxx1bvMkjpsyV9",
"type": "Entry",
"createdAt": "2021-11-10T13:57:10.882Z",
"updatedAt": "2021-11-11T10:58:39.480Z",
"environment": {
"sys": {
"id": "master",
"type": "Link",
"linkType": "Environment"
}
},
"publishedVersion": 13,
"publishedAt": "2021-11-10T14:06:51.393Z",
"firstPublishedAt": "2021-11-10T14:00:11.935Z",
"createdBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"updatedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"publishedCounter": 3,
"version": 23,
"publishedBy": {
"sys": {
"type": "Link",
"linkType": "User",
"id": "4123123123zOn3MkhuVB"
}
},
"contentType": {
"sys": {
"type": "Link",
"linkType": "ContentType",
"id": "hotelInfo"
}
}
},
"fields": {}
}
And in Contentful Admin area, all the fields of the post become empty.
Contentful documentation says:
Contentful doesn't merge changes made to content, so when updating
content, you need to send the entire body of an entry. If you update
content with a subset of properties, you will lose all existing
properties not included in that update.
You should always update resources in the following order:
Fetch current resource.
Make changes to the current resource.
Update the resource by passing the changed resource along with current version number.
This way no unseen changes are overridden and unexpected conflicts are
unlikely to occur.
Note: You can't update any of the sys property fields, including
sys.id.
...so, I guess, I'm doing everything right - taking the post, editing data and sending updated post back.
I tried editing my JSON data to send it without sys fields, but no luck.
I'm stuck, anyone has any ideas what should I proceed with?
Thanks to #whitep4nth3r I was able to solve the problem.
I needed to GET data from the same source I'm trying to PUT it to.
The Authorisation header needed to be replaced with the Personal access token used for PUT request.

set-base-url is getting always reset in backend section

I am trying to follow the link below for forwarding my api request to another base url, in Azure API management service, which i had created by importing apis from one azure function app.
https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-deploy-multi-region
my policies look like
<policies>
<inbound>
<base />
<choose>
<when condition="#(context.Request.OriginalUrl.Host.Contains("centralus-01.regional"))">
<set-backend-service base-url="https://apiservtmt.trafficmanager.net/api/" />
</when>
<when condition="#(context.Request.OriginalUrl.Host.Contains("southeastasia-01.regional"))">
<set-backend-service base-url="https://apiservt.trafficmanager.net/api/" />
</when>
<otherwise>
<set-backend-service base-url="https://apiservtmt.trafficmanager.net/api/" />
</otherwise>
</choose>
</inbound>
<backend>
<base />
</backend>
<outbound>
<base />
</outbound>
<on-error>
<base />
</on-error>
</policies>
but unfortunately even though i set the inbound policy to change the base url, post inbound policy is applied, the url is again getting reset to the original azure function that I had used to associate the apis with the api management.
u can see in the trace log, even after setting it to the base url of my desired choice its getting reset.
I tried every possible policy compbination, but still issue persists, please help.
api-inspector (0.322 ms)
{
"request": {
"method": "GET",
"url": "https://apiservapimt.azure-api.net/xtxs/sds/",
"headers": [
{
"name": "Sec-Fetch-Mode",
"value": "cors"
},
{
"name": "Ocp-Apim-Subscription-Key",
"value": "*****************"
},
{
"name": "Sec-Fetch-Site",
"value": "cross-site"
},
{
"name": "X-Forwarded-For",
"value": "188888888888888"
},
{
"name": "Cache-Control",
"value": "no-cache, no-store"
},
{
"name": "Content-Type",
"value": "text/plain;charset=UTF-8"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate,br"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.9,de;q=0.8,af;q=0.7"
},
{
"name": "Host",
"value": "apiservapimt.azure-api.net"
},
{
"name": "Referer",
"value": "https://apimanagement.hosting.portal.azure.net/apimanagement/Content/1.0.574.0/apimap//apimap-apis/index.html"
}
]
}
}
api-inspector (0.003 ms)
{
"configuration": {
"api": {
"from": "/",
"to": null,
"version": null,
"revision": "1"
},
"operation": {
"method": "GET",
"uriTemplate": "/jsdhakj/{surveyId}/"
},
"user": "-",
"product": "-"
}
}
cors (0.474 ms)
"Origin header was missing or empty and the request was classified as not cross-domain. CORS policy was not applied."
choose (0.010 ms)
{
"message": "Expression was successfully evaluated.",
"expression": "context.Request.OriginalUrl.Host.Contains(\"centralus-01.regional\")",
"value": false
}
choose (0.003 ms)
{
"message": "Expression was successfully evaluated.",
"expression": "context.Request.OriginalUrl.Host.Contains(\"southeastasia-01.regional\")",
"value": false
}
set-backend-service (0.005 ms)
{
"message": "Backend service URL was changed.",
"oldBackendServiceUrl": "",
"newBackendServiceUrl": "https://apiservtmt.trafficmanager.net/api/",
"request": {
"url": "https://apiservtmt.trafficmanager.net/api/adhsakjd/sds/"
}
}
**set-backend-service (0.005 ms)
"Backend 'apiservt' set to current."**
**Backend
(668.101 ms)↑ Back to top
set-backend-service (0.030 ms)
{
"message": "Backend service URL was changed.",
"oldBackendServiceUrl": "https://apiservtmt.trafficmanager.net/api/",
"newBackendServiceUrl": "https://apiservt.azurewebsites.net/api",
"request": {
"url": "https://apiservt.azurewebsites.net/api/surveys/sds/"
}
}**
set-header (0.019 ms)
{
"message": "Specified value was assigned to the header (see below).",
"header": {
"name": "x-functions-key",
"value": "hjwjrhewj"
}
}
forward-request (0.101 ms)
{
**"message": "Request is being forwarded to the backend service. Timeout set to 300 seconds",
"request": {
"method": "GET",
"url": "https://apiservt.azurewebsites.net/api/surveys/sds/",
"headers": [**
{
"name": "Host",
"value": "apiservt.azurewebsites.net"
},
{
"name": "Request-Id",
"value": "|9ab257c67368465b904f288b38b44118.c248de116fca4824_83e3e6e7."
},
{
"name": "Sec-Fetch-Mode",
"value": "cors"
},
{
"name": "Ocp-Apim-Subscription-Key",
"value": "wrewrwer"
},
{
"name": "Sec-Fetch-Site",
"value": "cross-site"
},
{
"name": "X-Forwarded-For",
"value": "shakjdhkjsa"
},
{
"name": "Cache-Control",
"value": "no-cache, no-store"
},
{
"name": "Content-Type",
"value": "text/plain;charset=UTF-8"
},
{
"name": "Accept",
"value": "*/*"
},
{
"name": "Accept-Encoding",
"value": "gzip,deflate,br"
},
{
"name": "Accept-Language",
"value": "en-US,en;q=0.9,de;q=0.8,af;q=0.7"
},
{
"name": "Referer",
"value": "https://apimanagement.hosting.portal.azure.net/apimanagement/Content/1.0.574.0/apimap//apimap-apis/index.html"
},
{
"name": "Request-Context",
"value": "appId=cid-v1:552f0e64-cebd-4730-b0b8-8cf0a0fa6caf"
},
{
"name": "x-functions-key",
"value": "shdsajdhskjf"
}
]
}
}
forward-request (667.950 ms)
{
"response": {
"status": {
"code": 401,
"reason": "Unauthorized"
},
"headers": [
{
"name": "Transfer-Encoding",
"value": "chunked"
},
{
"name": "Request-Context",
"value": "appId=cid-v1:552f0e64-cebd-4730-b0b8-8cf0a0fa6caf"
},
{
"name": "Content-Type",
"value": "application/json; charset=utf-8"
},
{
"name": "Date",
"value": "Tue, 24 Sep 2019 17:40:07 GMT"
}
]
}
}
Outbound
(0.146 ms)↑ Back to top
transfer-response (0.050 ms)
{
"message": "Response headers have been sent to the caller."
}
transfer-response (0.096 ms)
{
"message": "Response body streaming to the caller is complete."
}
its resolved now.
there was another set base url policy getting added at each operation level while importing the apis from the azure function. by removing these, picked the parent policy.

Request parameter RefreshType is not in correct format

I try to process my Azure Analysis Service Cube via logic APP:
In HTTP action I have the following body:
{
"inputs": {
"method": "POST",
"uri": "https://northeurope.asazure.windows.net/servers/xxx/models/xxx/refreshes",
"body": {
"refresh": {
"objects": [
{
"database": "ModelName",
"table": "TableName"
}
],
"type": "full"
}
},
"authentication": {
"type": "ActiveDirectoryOAuth",
"authority": "https://login.windows.net",
"tenant": "xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx",
"audience": "https://*.asazure.windows.net",
"clientId": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
"secret": "xxxxYjxx7y5xxskHxxxxxxznxxxxxxxxODX+xxxxxxxx"
}
}
}
After runningthis logic app, I get the following error:
{
"statusCode": 400,
"headers": {
"Strict-Transport-Security": "max-age=xxxxxx00; includeSubDomains",
"x-ms-root-activity-id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx",
"x-ms-current-utc-date": "2/7/2019 9:25:05 AM",
"x-ms-xmlaerror-extended": "Request parameter RefreshType is not in correct format.",
"Date": "Thu, 07 Feb 2019 09:25:05 GMT",
"Server": "Microsoft-HTTPAPI/2.0",
"Content-Length": "294",
"Content-Type": "application/json; charset=utf-8"
},
"body": {
"code": "BadRequest",
"subCode": 2,
"message": "Request parameter RefreshType is not in correct format.",
"timeStamp": "2019-02-07T09:25:05.4650388Z",
"httpStatusCode": 400,
"details": [
{
"code": "RootActivityId",
"message": "353ab74e-2793-4ee5-954f-819c851f4cb6"
},
{
"code": "Param1",
"message": "RefreshType"
}
]
}
}
Do you have any Idea, why I get this error message? and How can I solve this problem
Refresh Type is an ENUM which only takes values from below list, Can you make the type to Full and check if works:
Hope it helps.

BigQuery Error JSON_EXTRACT although jsonPath is valid

I'm trying to extract some information out of the httparchive har tables, which are provided for free.
Trouble is that BQ returns an error on a JSON_EXTRACT() function although a number of jsonPath validators are executing it perfectly (http://jsonpath.com/ or jsonpath.curiousconcept.com/).
This is my query:
SELECT
JSON_EXTRACT(payload,"$._host")AS host,
JSON_EXTRACT(payload,"$.request.headers[?(#.name=='Referer')]")AS referer,
url,
payload
FROM
[httparchive:har.2016_01_01_chrome_requests]
LIMIT
100
expected return based on the example payload json below would be
"http://www.echosdunet.net/"
This is the error:
Error: JSONPath parse error at: [?(#.name=='Referer')]
This is an example payload json:
{
"pageref": "page_1_0",
"startedDateTime": "2016-01-03T22:18:52.632+00:00",
"time": 452,
"request": {
"method": "GET",
"url": "http://disqus.com/embed/comments/?base=default&version=f3e1717b71e7256da258d3a504e56865&f=echosdunet&t_i=node%2F19849&t_u=http%3A%2F%2Fwww.echosdunet.net%2Fnode%2F19849&t_e=Accueil&t_d=Comparatif%20et%20test%20ADSL%20et%20fibre&t_t=Accueil&s_o=default&l=fr",
"headersSize": 650,
"bodySize": -1,
"cookies": [],
"headers": [{
"name": "Host",
"value": "disqus.com"
}, {
"name": "Connection",
"value": "keep-alive"
}, {
"name": "Accept",
"value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
}, {
"name": "Upgrade-Insecure-Requests",
"value": "1"
}, {
"name": "User-Agent",
"value": "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36 PTST/254"
}, {
"name": "Referer",
"value": "http://www.echosdunet.net/"
}, {
"name": "Accept-Encoding",
"value": "gzip, deflate, sdch"
}, {
"name": "Accept-Language",
"value": "en-US,en;q=0.8"
}],
"httpVersion": "1.1",
"queryString": [{
"name": "base",
"value": "default"
}, {
"name": "version",
"value": "f3e1717b71e7256da258d3a504e56865"
}, {
"name": "f",
"value": "echosdunet"
}, {
"name": "t_i",
"value": "node/19849"
}, {
"name": "t_u",
"value": "http://www.echosdunet.net/node/19849"
}, {
"name": "t_e",
"value": "Accueil"
}, {
"name": "t_d",
"value": "Comparatif et test ADSL et fibre"
}, {
"name": "t_t",
"value": "Accueil"
}, {
"name": "s_o",
"value": "default"
}, {
"name": "l",
"value": "fr"
}]
},
"response": {
"status": 200,
"statusText": "",
"headersSize": 1161,
"bodySize": 2017,
"headers": [{
"name": "Server",
"value": "nginx"
}, {
"name": "Content-Type",
"value": "text/html; charset=utf-8"
}, {
"name": "Content-Security-Policy",
"value": "script-src https://*.twitter.com:* https://api.adsnative.com/v1/ad.json *.adsafeprotected.com https://cas.criteo.com/delivery/0.1/napi.jsonp *.services.disqus.com:* http://referrer.disqus.com/juggler/ disqus.com http://*.twitter.com:* a.disquscdn.com https://referrer.disqus.com/juggler/ https://*.services.disqus.com:* *.moatads.com 'unsafe-eval' https://mobile.adnxs.com/mob https://ssl.google-analytics.com"
}, {
"name": "Link",
"value": "<http://a.disquscdn.com>;rel=preconnect,<http://a.disquscdn.com>;rel=dns-prefetch"
}, {
"name": "Cache-Control",
"value": "stale-if-error=3600, s-stalewhilerevalidate=3600, stale-while-revalidate=30, no-cache, must-revalidate, public, s-maxage=5"
}, {
"name": "p3p",
"value": "CP=\\DSP IDC CUR ADM DELi STP NAV COM UNI INT PHY DEM\\"
}, {
"name": "Timing-Allow-Origin",
"value": "*"
}, {
"name": "X-Content-Type-Options",
"value": "nosniff"
}, {
"name": "X-XSS-Protection",
"value": "1; mode=block"
}, {
"name": "Last-Modified",
"value": "Thu, 11 Jun 2015 13:30:36 GMT"
}, {
"name": "ETag",
"value": "W/\\lounge:view:3840102421.d93d9c4bc037078ffc811833ae267a6f.0\\"
}, {
"name": "Content-Encoding",
"value": "gzip"
}, {
"name": "Content-Length",
"value": "2017"
}, {
"name": "Accept-Ranges",
"value": "bytes"
}, {
"name": "Date",
"value": "Sun, 03 Jan 2016 22:18:51 GMT"
}, {
"name": "Age",
"value": "0"
}, {
"name": "Connection",
"value": "keep-alive"
}, {
"name": "Vary",
"value": "Accept-Encoding"
}],
"httpVersion": "1.1",
"redirectURL": "",
"content": {
"size": 2017,
"mimeType": "text/html"
},
"cookies": []
},
"cache": {},
"timings": {
"blocked": -1,
"dns": 202,
"connect": 32,
"ssl": -1,
"send": 0,
"wait": 108,
"receive": 110
},
"_ip_addr": "104.156.81.134",
"_method": "GET",
"_host": "disqus.com",
"_url": "/embed/comments/?base=default&version=f3e1717b71e7256da258d3a504e56865&f=echosdunet&t_i=node%2F19849&t_u=http%3A%2F%2Fwww.echosdunet.net%2Fnode%2F19849&t_e=Accueil&t_d=Comparatif%20et%20test%20ADSL%20et%20fibre&t_t=Accueil&s_o=default&l=fr",
"_responseCode": "200",
"_load_ms": "218",
"_ttfb_ms": "108",
"_load_start": "2632",
"_bytesOut": "652",
"_bytesIn": "3180",
"_objectSize": "2017",
"_cacheControl": "stale-if-error=3600, s-stalewhilerevalidate=3600, stale-while-revalidate=30, no-cache, must-revalidate, public, s-maxage=5",
"_contentType": "text/html",
"_contentEncoding": "gzip",
"_type": "3",
"_socket": "153",
"_score_cache": "-1",
"_score_cdn": "-1",
"_score_gzip": "100",
"_score_cookies": "-1",
"_score_keep-alive": "100",
"_score_minify": "-1",
"_score_combine": "-1",
"_score_compress": "-1",
"_score_etags": "-1",
"_is_secure": "0",
"_dns_ms": 202,
"_connect_ms": 32,
"_ssl_ms": "-1",
"_gzip_total": "3180",
"_gzip_save": "0",
"_minify_total": "0",
"_minify_save": "0",
"_image_total": "0",
"_image_save": "0",
"_cache_time": "-1",
"_dns_start": "2395",
"_dns_end": "2597",
"_connect_start": "2600",
"_connect_end": "2632",
"_ssl_start": "0",
"_ssl_end": "0",
"_initiator": "http://echosdunet.disqus.com/embed.js?_=1451859532217",
"_initiator_line": "16",
"_initiator_column": "8205",
"_server_count": "4",
"_server_rtt": "32",
"_client_port": "62284",
"_jpeg_scan_count": "0",
"_full_url": "http://disqus.com/embed/comments/?base=default&version=f3e1717b71e7256da258d3a504e56865&f=echosdunet&t_i=node%2F19849&t_u=http%3A%2F%2Fwww.echosdunet.net%2Fnode%2F19849&t_e=Accueil&t_d=Comparatif%20et%20test%20ADSL%20et%20fibre&t_t=Accueil&s_o=default&l=fr",
"_score_progressive_jpeg": -1,
"_body": true,
"_load_end": 2850,
"_ttfb_start": "2632",
"_ttfb_end": 2740,
"_download_start": 2740,
"_download_end": 2850,
"_download_ms": 110,
"_all_start": "2395",
"_all_end": 2850,
"_all_ms": 452,
"_index": 55,
"_number": 56,
"_body_url": "/response_body.php?test=160101_10_KZQ1&run=1&cached=0&request=56"
}
Running the query without the referer jsonPath works fine:
SELECT
JSON_EXTRACT(payload,"$._host")AS host,
url,
payload
FROM
[httparchive:har.2016_01_01_chrome_requests]
LIMIT
100
Meantime, try below (for BigQuery Standard SQL)
#standardSQL
CREATE TEMPORARY FUNCTION CUSTOM_JSON_EXTRACT(json STRING, key String)
RETURNS string
LANGUAGE js AS """
try {
var parsed = JSON.parse(json);
for (i = 0; i < parsed.length; i++) {
if (parsed[i].name == key) return parsed[i].value
}
} catch (e) {}
return null;
""";
SELECT
JSON_EXTRACT(payload,"$._host")AS host,
CUSTOM_JSON_EXTRACT(JSON_EXTRACT(payload, "$.request.headers"), 'Referer') AS referer,
url,
payload
FROM
`httparchive.har.2016_01_01_chrome_requests`
LIMIT
100
JSON_EXTRACT and JSON_EXTRACT_SCALAR do not support the complete set of JSONPath elements. You can read more in the documentation. I would suggest using standard SQL since that implementation of the JSON functions has better error messages.
If you are interested in additional kinds of JSONPath elements, you can consider submitting a feature request.