Data Lake Analytics job failed with the following error with input as json file. Looks like a json formatting error. Any help is greatly appreciated.
The json file was generated by stream analytics with data from IOT hub as input.
{
"jobId": "https://iotdla4c2c5a3b.azuredatalakeanalytics.net/jobLink/4eddd130-e0fc-45d4-8b3a-179b0e49f6c1",
"errors": {
"diagnosticCode": 195887158,
"severity": "Error",
"component": "RUNTIME",
"source": "User",
"errorId": "E_RUNTIME_USER_UNHANDLED_EXCEPTION_FROM_USER_CODE",
"message": "An unhandled exception from user code has been reported when invoking the method 'Extract' on the user type 'Microsoft.Analytics.Samples.Formats.Json.JsonExtractor'",
"description": "Unhandled exception from user code: \"Additional text encountered after finished reading JSON content: {. Path '', line 2, position 0.\"\nThe details includes more information including any inner exceptions and the stack trace where the exception was raised.",
"resolution": "Make sure the bug in the user code is fixed.",
"helpLink": "",
"details": "==== Caught exception Newtonsoft.Json.JsonReaderException\n\n at Newtonsoft.Json.JsonTextReader.Read()\r\n at Microsoft.Analytics.Samples.Formats.Json.JsonExtractor.d__2.MoveNext()\r\n at ScopeEngine.RunUdoCodeExceptionhandled >(ScopeTypedManagedHandle* className, SByte* methodName, * code) in d:\data\yarnnm\local\usercache\9bb5486e-cb76-464b-a635-28243eb11fb0\appcache\application_1556746128054_0157\container_e391_1556746128054_0157_01_000001\wd\managed.h:line 2034",
"userData": ""
}
}
DataLake stores json in two different formats. 1. line Feed. 2 Array. Changing the format to Array solved this problem
Related
A webhook call fails, how do I see the detailed logs?
The testing webhooks page describes that the errors are put in output.webhook_error.<result_variable>. For example if your result variable is webhook_result_1 then the output is in output.webhook_error.webhook_result_1.
As described in the documentation, if an error happens then in the "Assistant responds" section of the dialog node editor, the "If assistant recognizes" anything_else should be used, so you can put in there The callout generated this error: <? output.webhook_error.webhook_result_1 ?>..
This will show more information. For example if I configure an API to throw an Error, I will get back:
The callout generated this error: {"response_code":400,"message":"Webhook call was not successful. Response code is [400].","response_body":{"code":"cff11d18150e8018d5e304ac3fc35c94","error":"There was an error processing your request."},"content_type":"application/json"}.
If this error happens when the Bot is being used live, then in the logs the webhook response body will be captured. If you use the Try It Out panel in the UI then the logs are not captured.
In the Assistant UI for the Skill in the Analytics tab the conversation is shown. In the "User conversations" part of the UI the text of the conversation is captured, but you will only see the webhooks error if you add output as described above.
The API can be used to fetch logs for a workspace or skill which include the webhook error details. See List Log Events in a Workspace. If you are using a Skill then you can find out the URL to use from the Assistant UI. On the Skills list page, on the particular skill click the ... menu and select View API Details. The Legacy v1 workspace URL: shows the workspace ID to use.
For example if the Legacy v1 workspace URL: is https://gateway.watsonplatform.net/assistant/api/v1/workspaces/ce13f844-c3a7-4f36-97c6-a0ac704024a6/message the URL to use to fetch logs is https://gateway.watsonplatform.net/assistant/api/v1/workspaces/ce13f844-c3a7-4f36-97c6-a0ac704024a6/logs?version=2020-04-01
In the workspace log there will be the details of the user conversation where a webhook error happened in webhook_error:
"output": {
"generic": [
{
"response_type": "text",
"text": ""
},
{
"response_type": "text",
"text": "The callout generated this error: {\"response_code\":400,\"message\":\"Webhook call was not successful. Response code is [400].\",\"response_body\":{\"code\":\"cff11d18150e8018d5e304ac3fc35c94\",\"error\":\"There was an error processing your request.\"},\"content_type\":\"application/json\"}."
},
{
"response_type": "text",
"text": "Customer email is <? context.webhook_result_1.message ?>."
}
],
"text": [
"",
"The callout generated this error: {\"response_code\":400,\"message\":\"Webhook call was not successful. Response code is [400].\",\"response_body\":{\"code\":\"cff11d18150e8018d5e304ac3fc35c94\",\"error\":\"There was an error processing your request.\"},\"content_type\":\"application/json\"}.",
"Customer email is <? context.webhook_result_1.message ?>."
],
"nodes_visited": [
"slot_2_1520179906877",
"handler_3_1520179906877",
"node_1_1520179877410",
"node_9_1530805081298",
"response_5_1604940050659",
"node_7_1520180645829"
],
"webhook_error": {
"webhook_result_1": {
"response_code": 400,
"message": "Webhook call was not successful. Response code is [400].",
"response_body": {
"code": "cff11d18150e8018d5e304ac3fc35c94",
"error": "There was an error processing your request."
},
"content_type": "application/json"
}
},
"nodes_visited_details": [
Everything appears correct within my Logic App:
Extracting data from CosmosDB, Pushing to CRM and then returning this data to CosmosDB.
However, I keep getting this error 400.
Here is my code:
Extracting and parsing data from CosmosDB
Pushing data that I have in CosmosDB that I wish to go into CRM
Returning data with newly replaced CRMid back into the original CosmosDB container
However I keep getting this error:
Body:
{
"_ems_leadpm_value": "test
"_ems_leadpmforfeature_value": "",
"_ems_productname_value": "",
"_ownerid_value": "test",
"ems_featurdescription": "test.",
"ems_featurename": "test
"ems_keywords": ""
}
Specific error: 400, what does position 8 mean? it seems the program is wanting me to add parentheses or a comma? but where should this be added? I am still trying to research this and am unsure.
{
"status": 400,
"message": "')' or ',' expected at position 8 in '(test)'.",
"error": {
"message": "')' or ',' expected at position 8 in '(test)'."
},
"source": "test.crm.dynamics.com",
"errors": [],
"debugInfo": "clientRequestId: 1111111"
}
Any suggestion or advice is appreciated
I'm requesting a list of possible stock ticker symbols from the Yahoo Stock Lookup API, I used the CORS Anywhere service to avoid problems. Unfortunately the original json response from the Yahoo API gets formatted as 'text' in the CORS anywhere 'error' property.
I used this previously asked question as a reference on how to access nested properties.
this is what the json reponse looks like:
{
"headers": {
"normalizedNames": {},
"lazyUpdate": null
},
"status": 200,
"statusText": "OK",
"url": "https://cors-anywhere.herokuapp.com/http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=Amd®ion=1&lang=en&callback=YAHOO.Finance.SymbolSuggest.ssCallback",
"ok": false,
"name": "HttpErrorResponse",
"message": "Http failure during parsing for https://cors-anywhere.herokuapp.com/http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=Amd®ion=1&lang=en&callback=YAHOO.Finance.SymbolSuggest.ssCallback",
"error": {
"error": {
"line": 9848,
"column": 56,
"sourceURL": "capacitor://localhost/vendor-es2015.js"
},
"text": "YAHOO.Finance.SymbolSuggest.ssCallback({\"ResultSet\":{\"Query\":\"Amd\",\"Result\":[{\"symbol\":\"AMD\",\"name\":\"Advanced Micro Devices, Inc.\",\"exch\":\"NMS\",\"type\":\"S\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Equity\"},{\"symbol\":\"DOX\",\"name\":\"Amdocs Limited\",\"exch\":\"NMS\",\"type\":\"S\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Equity\"},{\"symbol\":\"AMDVX\",\"name\":\"American Century Mid Cap Value Fund R6 Class\",\"exch\":\"NAS\",\"type\":\"M\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Fund\"},{\"symbol\":\"AMDWX\",\"name\":\"Amana Mutual Funds Trust Developing World Fund Investor\",\"exch\":\"NAS\",\"type\":\"M\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Fund\"},{\"symbol\":\"AMDAX\",\"name\":\"AMIDEX35 Israel Fund Class A\",\"exch\":\"NAS\",\"type\":\"M\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Fund\"},{\"symbol\":\"AMDEX\",\"name\":\"AMIDEX35 Israel Mutual Fund\",\"exch\":\"NAS\",\"type\":\"M\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Fund\"},{\"symbol\":\"AMDCX\",\"name\":\"AMIDEX35 Israel Fund Class C\",\"exch\":\"NAS\",\"type\":\"M\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Fund\"},{\"symbol\":\"AMDRX\",\"name\":\"American Beacon Mid-Cap Value Fund R6 Class\",\"exch\":\"NAS\",\"type\":\"M\",\"exchDisp\":\"NASDAQ\",\"typeDisp\":\"Fund\"},{\"symbol\":\"AMDLY\",\"name\":\"Amada Holdings Co., Ltd.\",\"exch\":\"PNK\",\"type\":\"S\",\"exchDisp\":\"OTC Markets\",\"typeDisp\":\"Equity\"}]}});"
}
}
This is what I tried:
return this.yahooResult.subscribe(data => {return data['error']['text']['ResultSet']['Result']['symbol']});
I'm trying to access "symbol" : "AMD" from the Yahoo API response that is nested in the "text" property of the CORS repsonse.
I don't get much information from the error:
⚡️ [error] - ERROR
You don't have json there, you have a javascript snippet:
YAHOO.Finance.SymbolSuggest.ssCallback()
With some escaped text passed into it.
It's not a case of parsing this, its a case of fixing the error.
I think you are using the wrong API call. It has a callback parameter in it which is wrapping the reply in a function and breaking cors-anywhere's ability to parse the response.
The url you should be using is:
http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=Amd®ion=1&lang=en
Not:
http://d.yimg.com/autoc.finance.yahoo.com/autoc?query=Amd®ion=1&lang=en&callback=YAHOO.Finance.SymbolSuggest.ssCallback
Then just plain JSON gets returned and cors-anywhere should be able to parse it.
I've got a API stage that's NOT using "Lambda Proxy integration" which has a Lambda function passing an error.
In the mapping template I have this:
$input.path("$.errorMessage")
Which results in the output of this:
{
"headers": {
"apiVersion": "20190218.1",
"isTesting": true
},
"body": {
"statusCode": 503,
"status": "Service Unavailable",
"title": "One or more of our data providers are currently offline for scheduled maintenance"
}
}
The header values are mapped to template headers and pull through correctly, however I need the body to transform to this:
{
"statusCode": 503,
"status": "Service Unavailable",
"title": "One or more of our data providers are currently offline for scheduled maintenance"
}
Whatever I have tried, body always returns as a blank string, an empty body, or an invalid JSON.
This is the closest I've got but it returns an invalid JSON:
$util.parseJson($input.path("$.errorMessage")).body
Result (comes back with no quotes):
{statusCode=503, status=Service Unavailable, title=One or more of our data providers are currently offline for scheduled maintenance}
Is it possible to do what I'm after? I can't find a reverse for $util.parseJson (i.e, stringify).
Thanks!
I think the original poster has probably moved on in the past 11 months, but in case anyone else stumbles across this question, $input.json('$.errorMessage.body') should work.
I am writing a REST API and I have stumbled upon a problem. What is the best way to return the validation errors.
Until now I have been returning the error messages dumped into a general error code (let's say bad request for example)
{
"status": 400,
"error": {
"code": 1, // General bad request code
"message": [
"The Key \"a\" is missing",
"The Key \"b\" is missing",
"The Key \"c\" is missing",
"Incorrect Format for field \"y\""
]
}
)
I have researched a little more about how should a good API response should look like and I thought of the following options:
Stop at the first encountered error and return a response with the specific error code
{
"status": 400, //Same as the HTTP header returned
"error" {
"code": 1, // Specific field validation error code
"message": "Field \"x\" is missing from the array structure",
"developer_message": "The request structure must contain the following fields {a,b,c{x,y,z}}",
"more_info" => "www.api.com/help/errors/1"
}
)
Parse all the request data and return multiple field validation errors.
{
"status": 400,
"error": {
"code": 1 //General bad Request code
"message": "Bad Request",
"developer_message": "Field validation errors."
"more_info": "www.api.com/help/errors/1",
"error_details": {
0: {
"code": 2 // Specific field validation error code
"message": "Field \"x\" is missing from the array structure",
"developer_message": "The request structure must contain the following fields {a,b,c{x,y,z}}",
"more_info": "www.api.com/help/errors/2"
},
1: {
"code": 3 // Specific field validation error code
"message": "Incorrect Format for field \"y\"",
"developer_message": "The field \"y\" must be in the form of \"Y-m-d\"",
"more_info": "www.api.com/help/errors/3"
}
}
}
}
In my opinion option 2 would be the right way (it gives more useful information to the developers/end users and the server load might be lower(less requests/no need to revalidate valid data/no need to compute the signature and authenticate the user)), but I am wandering what are the best practices, and if there is another way to treat this kind of problems.
Also i think option 1 is still valid if i get a single fatal error in the flow of the script.(not validation errors)
Please note that the code is just a simple array just so it is easier to follow. The response format will be JSON or XML.
Let's look to Facebook's Graph API. That is hit hard, and a great many errors are most likely generated. Here is what Facebook returns on an API error:
{
"error": {
"message": "Message describing the error",
"type": "OAuthException",
"code": 190,
"error_subcode": 460,
"error_user_title": "A title",
"error_user_msg": "A message"
}
}
They try to make the Graph API as useful as possible, but they seem return a specific error with a code and a subcode (Ref). The fact that each error has its own code means it's easier to search for said code or message as a starting point for debugging. That's probably why they don't accumulate error messages in their official error response. If it's good enough and convenient for Facebook, it's probably good enough for us.
Sample error responses:
{
"error": {
"message": "(#200) Must have a valid access_token to access this endpoint",
"type": "OAuthException",
"code": 200
}
}
and
"error": {
"message": "(#604) Your statement is not indexable. The WHERE clause must contain
an indexable column. Such columns are marked with * in the tables linked from
http://developers.facebook.com/docs/reference/fql ",
"type": "OAuthException",
"code": 604
}
Then there is JSend which "is a specification that lays down some rules for how JSON responses from web servers should be formatted." Their goal is:
There are lots of web services out there providing JSON data, and each has its own way of formatting responses. Also, developers writing for JavaScript front-ends continually re-invent the wheel on communicating data from their servers. While there are many common patterns for structuring this data, there is no consistency in things like naming or types of responses. Also, this helps promote happiness and unity between backend developers and frontend designers, as everyone can come to expect a common approach to interacting with one another.
Here is a sample error message:
{
"status" : "fail",
"data" : { "title" : "A title is required" }
}
It looks like Facebook and this group trying to set like an industry standard are opting for your choice #1.
Bounty Question
In response to the bounty request of "if anyone went #2 and maybe has any improvements on it?", there is a design pattern from Pragmatic RESTful API that states:
Validation errors will need a field breakdown. This is best modeled by using a fixed top-level error code for validation failures and providing the detailed errors in an additional errors field, like so:
{
"code" : 1024,
"message" : "Validation Failed",
"errors" : [
{
"code" : 5432,
"field" : "first_name",
"message" : "First name cannot have fancy characters"
},
{
"code" : 5622,
"field" : "password",
"message" : "Password cannot be blank"
}
]
}
I have used #2 myself a couple times. Is it better than #1? I think that depends on what your API is being used for.
I like #2 because it gives a developer who is testing the API with some test calls a quick overview of all the errors/mistakes he made in a request, so he knows immediately which errors/mistakes he has to fix to make that request valid. If you return the errors one by one (like in #1) you have to keep retrying the request and cross fingers hoping it will be valid this time.
But as I said #2 is very useful for developers, but the reasons don't really apply to end users. End users don't usually care how it's implemented. Whether the software is doing 1 request which returns 5 errors or 5 subsequent requests which return 1 error each.
As long as it's handled well in the client, the end user shouldn't notice the difference. How to handle that of course very much depends on what the client actually is.
Next to speeding up the development, another benefit of #2 (in production) is that it requires less requests to be made, which of course decreases the server load.
I would like to know if anyone went #2 and maybe have any improvements on it so I opened a bounty.
Sure there are improvements to be made. As it is, there is some data in the body that can be omitted.
{
"status": 400,
"error": {
"code": 1 //General bad Request code
"message": "Bad Request",
"developer_message": "Field validation errors."
"more_info": "www.api.com/help/errors/1",
"error_details": {
0: {
"code": 2 // Specific field validation error code
"message": "Field \"x\" is missing from the array structure",
"developer_message": "The request structure must contain the following fields {a,b,c{x,y,z}}",
"more_info": "www.api.com/help/errors/2"
},
1: {
(
"code": 3 // Specific field validation error code
"message": "Incorrect Format for field \"y\"",
"developer_message": "The field \"y\" must be in the form of \"Y-m-d\"",
"more_info": "www.api.com/help/errors/3"
)
}
)
With HTTP responses, the status code should not go in the body, but in the header. This means that "status": 400 and "message": "Bad Request" can be omitted here. 400 should be the response's status code, and 400 means Bad Request. It's a HTTP standard and doesn't have to be explained in the response. Also "developer_message": "Field validation errors." is kind of a duplicate, since the specific errors are already included in each seperate error, so we could leave that out.
That leaves
{
"error": {
"code": 1 //General bad Request code
"more_info": "www.api.com/help/errors/1",
"error_details": {
0: {
"code": 2 // Specific field validation error code
"message": "Field \"x\" is missing from the array structure",
"developer_message": "The request structure must contain the following fields {a,b,c{x,y,z}}",
"more_info": "www.api.com/help/errors/2"
},
1: {
(
"code": 3 // Specific field validation error code
"message": "Incorrect Format for field \"y\"",
"developer_message": "The field \"y\" must be in the form of \"Y-m-d\"",
"more_info": "www.api.com/help/errors/3"
)
}
)
"code": 1 //General bad Request code
"more_info": "www.api.com/help/errors/1",
These 2 lines don't really make sense anymore now. They are also not required, since each error has it's own code and info link, so we might strip these lines as well, leaving this
{
"error": {
"error_details": {
0: {
"code": 2 // Specific field validation error code
"message": "Field \"x\" is missing from the array structure",
"developer_message": "The request structure must contain the following fields {a,b,c{x,y,z}}",
"more_info": "www.api.com/help/errors/2"
},
1: {
(
"code": 3 // Specific field validation error code
"message": "Incorrect Format for field \"y\"",
"developer_message": "The field \"y\" must be in the form of \"Y-m-d\"",
"more_info": "www.api.com/help/errors/3"
)
}
)
The 400 status code already indicates there was an error, so you don't have to indicate "error": {error details} anymore, because we already know there was an error. The list of errors can simply become the root object:
[
{
"code": 2//Specificfieldvalidationerrorcode
"message": "Field \"x\" is missing from the array structure",
"developer_message": "The request structure must contain the following fields {a,b,c{x,y,z}}",
"more_info": "www.api.com/help/errors/2"
},
{
"code": 3//Specificfieldvalidationerrorcode
"message": "Incorrect Format for field \"y\"",
"developer_message": "The field \"y\" must be in the form of \"Y-m-d\"",
"more_info": "www.api.com/help/errors/3"
}
]
So all that's left in the body now is simply a list of errors.
The status code is specified in the response header.
The details are specified in the response body.
I recently worked against a Rest API that would return multiple warnings or errors in the results. Starting from your Sample #2, I would modify it as follows:
{
"status": 400,
"results" : null,
"warnings": {
0: {
// Build a warning message here, sample text to show concept
"code": 1 // Specific field validation error code
"message": "It is no longer neccessary to put .js on the URL"
}
}
"errors": {
0: {
"code": 2 // Specific field validation error code
"message": "Field \"x\" is missing from the array structure"
"developer_message": "The request structure must contain the following fields {a,b,c{x,y,z}}",
},
1: {
"code": 3 // Specific field validation error code
"message": "Incorrect Format for field \"y\"",
"developer_message": "The field \"y\" must be in the form of \"Y-m-d\""
}
}
}
This would provide you the ability to provide results, with multiple warnings or errors as needed in your response.
And yes, this does have some bloat in the structure, but it also provides an easy interface for a developer to always get their data back in the same structure.
I would also remove the following items as IMHO they should be in the API docs (how to find help using an error code) instead of on every error:
"more_info": "www.api.com/help/errors/2"
"more_info": "www.api.com/help/errors/3"
Along those same lines, I'm not sure if you need both the message and developer_message. They appear to be redundant and as if you are trying to provide user error messages from the API when the caller failed to provide data correctly.
First of all you would have provided documentation for the Rest API methods for the customers. So it is expected from the customer/developer to provide valid data for the parameters.
Now having said that #1 is the best way to do Rest API. A developer's responsibility is to reduce server usage to the maximum possible extent. So if you encounter any fatal error, Construct a response with the corresponding error code and error message and return it.
Moreover we cannot be sure that there are more errors after the one we encountered. So there is no point in parsing the rest of the data. It will not work well considering the worst case.
Personally I would give the users less detail and either dump errors needed to the developers in a database log table or system logs. Due to the fact you are using JSON and that is most common on Apache servers and your code looks likely to be php (but your code example with curly braces could be a number of languages originating from PASCAL eg. C, C# PERL, PHP, CSharp). Here is how to put output custom errors to the system logs if you do not already know how in php http://php.net/manual/en/function.syslog.php . If your are using a rarer configuration IIS with JSON and CSharp there are .NET libraries to do similar too. If you give your users too much information when an error occurs you are also giving a hacker in the future a way of probing your website.
API's are not for humans. Therefore you don't need to return detailed error texts. You can even return an error code which means "missing parameter". Just don't forget to document it well.