I am new to Python/Scala. Trying to read/parse a JSON file from dataLake using Python and Scala but unsuccessful. Is there something wrong with the .JSON schema. I have attached the JSON schema and the data frame screenshots. Any help would be highly appreciated.
{
"result": [
{
"instance": {
"link": "https://service-now.com/api/now/table/assessment_instance/3d2a2c44d720058ba859e03d2",
"value": "3d2a2c44d720058ba859e03d2"
},
"string_value": "Very Good",
"assign_to_order": "",
"instance_question": {
"link": "https://service-now.com/api/now/table/assessment_instance_question/022a2c44d720058ba859e03d7",
"value": "022a2c44d720058ba859e03d7"
},
"template_definition": "",
"metric_definition": {
"link": "https://service-now.com/api/now/table/metric_definition/7758eb387233200dbba82cb0bfb",
"value": "7758eb387233200dbba82cb0bfb"
},
"sys_updated_on": "207-07-0 02:3:3",
"scaled_value": "4",
"metric": {
"link": "https://service-now.com/api/now/table/metric/542a2c44d720058ba859e03a4",
"value": "542a2c44d720058ba859e03a4"
},
"sys_created_on": "203-2-04 2:34:08",
"sys_domain": {
"link": "https://service-now.com/api/now/table/sys_user_group/global",
"value": "global"
},
"weighted_value": "0.8",
"source_id": {
"link": "https://service-now.com/api/now/table/type/502a2c44d720",
"value": "502a2c44d720058ba859e03a3"
},
"normalized_value": ".33",
"source_table": "metric_type",
"user": "",
"sys_created_by": "admin"
}
]
}
val asmt_metric_result = spark.read.option("multiline", "true")
.json(s"c:/servicenow/asmt_metric_result/2021/09/28/08/32/asmt_metric_result_000000.json")
asmt_metric_result.show(false)
Related
I need to standardize the object response key/values so that they are easier to parse/traverse using the tool I'm integrating.
Starting with the following JSON:
{
"status": true,
"body": {
"phone": "+1 937-830-1167",
"address": "2323 kuhku",
"linkedin": "uhku",
"twitter": "uhukh",
"education": "weeww",
"work_experience": "wewaew",
"write_something_about_you": "yugtyt",
"why_you_think_you_are_good_for_this_job": "kuhhuk",
"write_your_assignment_question": "kuhghuhghj",
"upload_your_attachment": null,
"upload_your_resume_here": null
}
}
Using Dart, what would be the best way to reformat as shown?
{
"status": true,
"body": {
"answers":[
{
"label": "phone",
"answer":"+1 937-830-1167"
},
{
"label": "address",
"answer":"2323 kuhku"
},
{
"label": "linkedin",
"answer": "uhku"
},
{
"label": "twitter",
"answer": "uhku"
},
{
"label": "education",
"answer": "uhku"
},
{
"label": "work_experience",
"answer": "uhku"
},
{
"label": "write_something_about_you",
"answer": "uhku"
},
{
"label": "why_you_think_you_are_good_for_this_job",
"answer": "uhku"
},
{
"label": "write_your_assignment_question",
"answer": "uhku"
},
{
"label": "upload_your_attachment",
"answer": "uhku"
},
{
"label": "write_something_about_you",
"answer": "upload_your_resume_here"
}
]
}
}
I'm somewhat limited by the tool I'm using so this will make it much easier to parse the JSON object with JSON Path as needed.
Looks like something that should be easily doable as:
var json = ... your json ...;
var result = {
"status": json["status"],
"body": [for (var e in json["body"].entries)
{"label": e.key, "answer": e.value}
]
};
Create a new JSON object with the same "status" and a "body" which is a list instead of a map, and for each entry in the original map, create a JSON object with a "label" and "answer" taken from the key and value of the map entry.
I want to sort the values in csv (or google sheet)with an API. But my knowledge of JSON=0. I'm not sure how to retrieve the right data.
Also i dont know how to sort JSON (xD)
import json
import requests
import csv
api_key = "https://api.blablablbla"
r = requests.get('https://api.blablablbla')
packages_json = r.json()
The sorting should by header name: projectnumber, project name etc. and the rows should be the "value" corrosponding the header. I feel like the imput of the data is not perfect. The input is as follows(parsed):
{
"results": [
{
"submittedAt": 1614950496814,
"values": [
{
"name": "projectnumber?",
"value": "123",
"objectTypeId": "0-1"
},
{
"name": "project name?_",
"value": "Project X",
"objectTypeId": "0-1"
},
{
"name": "email",
"value": "abc#123.com",
"objectTypeId": "0-1"
},
{
"name": "firstname",
"value": "James",
"objectTypeId": "0-1"
},
{
"name": "lastname",
"value": "Bond",
"objectTypeId": "0-1"
},
{
"name": "Are you available?_",
"value": "Maybe later",
"objectTypeId": "0-1"
},
{
"name": "Send reminder?_",
"value": "Yes",
"objectTypeId": "0-1"
},
{
"name": "form filled in by",
"value": "Jimmy",
"objectTypeId": "0-1"
}
],
"pageUrl": "https://share.hsforms.com/.........."
},
The second 1:
{
"submittedAt": 1614943909980,
"values": [
{
"name": "projectnumber?",
"value": "1234",
"objectTypeId": "0-1"
},
{
"name": "project name?",
"value": "XYZ",
"objectTypeId": "0-1"
},
{
"name": "email",
"value": "bca#bobs.nl",
"objectTypeId": "0-1"
},
{
"name": "firstname",
"value": "Jan",
"objectTypeId": "0-1"
},
{
"name": "lastname",
"value": "de Vries",
"objectTypeId": "0-1"
},
{
"name": "Are you available?",
"value": "maybe later",
"objectTypeId": "0-1"
},
{
"name": "Send reminder",
"value": "yes",
"objectTypeId": "0-1"
}
],
"pageUrl": "https://share.hsforms.com/........."
},
Help!
I think in this solution but probably if I knew the context the aswer would be more acurate. In first place you should receive the parameters of where you want to insert the values. For example, you use the google sheet API the json will be something like this:
{
"connection":{
"workbookID": "ABC",
"sheetName": "ABC"
}
"values": [
{
"name": "column1",
"values": ["rowA", "rowB", "rowC"]
},
{
"name": "column2",
"values": ["rowA", "rowB", "rowC"]
}]
}
In the same way, you can change the connections to get the CSV parameters and parce the values on it.
Then you can read the values as arrays and insert them without complication.
I'm new to Groovy. But, I can try with some simple code lines to extract the data.
I have read the from Parse JSON using groovy script (using JsonSlurper). But it didn't help in my case.
My code line:
{
"errors": false,
"address_data": [
{
"address_id": "567",
"township": {
"id": "41079",
"name": "Test Data"
},
"city": {
"id": "1622",
"name": "Test City"
},
"region": {
"id": "663",
"name": "Metro Test"
},
"stock_source_code": "Test",
"is_default_address": false
},
{
"address_id": "45444",
"township": {
"id": "41079",
"name": "Test Test"
},
"city": {
"id": "1622",
"name": "Test City"
},
"region": {
"id": "663",
"name": "Metro Test Taguig"
},
"is_default_address": true
},
{
"address_id": "45444",
"township": {
"id": "888888",
"name": "Apas"
},
"city": {
"id": "432",
"name": "Test City"
},
"region": {
"id": "591",
"name": "Test Cebu"
},
"stock_source_code": "testce",
"is_default_address": false
}
]
}
My code lines:
def response = "data_above";
def object = new JsonSlurper().parseText(response);
def errors = object.errors
if (errors == false) { //could access object.errors property
log.info "Checking condition"
def addressData = object.address_data // Cannot get the data
}
How can I get the address_data property?
I want to index & search nested json in solr. Here is my json code
{
"id": "44444",
"headline": "testing US",
"generaltags": [
{
"type": "person",
"name": "Jayalalitha",
"relevance": "0.334",
"count": 1
},
{
"type": "person",
"name": "Kumar",
"relevance": "0.234",
"count": 1
}
],
"socialtags": {
"type": "SocialTag",
"name": "US",
"importance": 2
},
"topic": {
"type": "Topic",
"name": "US",
"score": "0.936"
}
}
When I try to Index, I'm getting the error "Error parsing JSON field value. Unexpected OBJECT_START"
When we tried to use Multivalued Field & index, we couldn't able to search using the multivalued field? Its returning "Undefined Field"
Also Please advice if I need to do any changes in schema.xml file?
You are nesting child documents within your document. You need to use the proper syntax for nested child documents in JSON:
[
{
"id": "1",
"title": "Solr adds block join support",
"content_type": "parentDocument",
"_childDocuments_": [
{
"id": "2",
"comments": "SolrCloud supports it too!"
}
]
},
{
"id": "3",
"title": "Lucene and Solr 4.5 is out",
"content_type": "parentDocument",
"_childDocuments_": [
{
"id": "4",
"comments": "Lots of new features"
}
]
}
]
Have a look at this article which describes JSON child documents and block joins.
Using the format mentioned by #qux you will face "Expected: OBJECT_START but got ARRAY_START at [16]",
"code": 400
as when JSON starting with [....] will parsed as a JSON array
{
"id": "44444",
"headline": "testing US",
"generaltags": [
{
"type": "person",
"name": "Jayalalitha",
"relevance": "0.334",
"count": 1
},
{
"type": "person",
"name": "Kumar",
"relevance": "0.234",
"count": 1
}
],
"socialtags": {
"type": "SocialTag",
"name": "US",
"importance": 2
},
"topic": {
"type": "Topic",
"name": "US",
"score": "0.936"
}
}
The above format is correct.
Regarding searching. Kindly use the index to search for the elements of the JSON array.
The workaround for this can be keeping the whole JSON object inside other JSON object and the indexing it
I was suggesting to keep the whole data inside another JSON object. You can try the following way
{
"data": [
{
"id": "44444",
"headline": "testing US",
"generaltags": [
{
"type": "person",
"name": "Jayalalitha",
"relevance": "0.334",
"count": 1
},
{
"type": "person",
"name": "Kumar",
"relevance": "0.234",
"count": 1
}
],
"socialtags": {
"type": "SocialTag",
"name": "US",
"importance": 2
},
"topic": {
"type": "Topic",
"name": "US",
"score": "0.936"
}
}
]
}
see the syntax in http://yonik.com/solr-nested-objects/
$ curl http://localhost:8983/solr/demo/update?commitWithin=3000 -d '
[
{id : book1, type_s:book, title_t : "The Way of Kings", author_s : "Brandon Sanderson",
cat_s:fantasy, pubyear_i:2010, publisher_s:Tor,
_childDocuments_ : [
{ id: book1_c1, type_s:review, review_dt:"2015-01-03T14:30:00Z",
stars_i:5, author_s:yonik,
comment_t:"A great start to what looks like an epic series!"
}
,
{ id: book1_c2, type_s:review, review_dt:"2014-03-15T12:00:00Z",
stars_i:3, author_s:dan,
comment_t:"This book was too long."
}
]
}
]'
supported from solr 5.3
why this error
xhr: [object XMLHttpRequest]
textStatus: parsererror
errorThrown: Invalid JSON: [ { "id": "10000000", "label": "10000000" }, { "id": "10000001", "label": "10000001" }, { "id": "10000003", "label": "10000003" }, { "id": "10000005", "label": "10000005" }, { "id": "10000006", "label": "10000006" } ]
Copied and pasted this:
[ { "id": "10000000", "label": "10000000" }, { "id": "10000001", "label": "10000001" }, { "id": "10000003", "label": "10000003" }, { "id": "10000005", "label": "10000005" }, { "id": "10000006", "label": "10000006" }]
to JSONlint and it says it's valid.
Can you post the original JSON if that is the original then the it is missing a root key
var textStatus = {"keys":[{"id":"...", "label":"..."},{"id":"...","label":"..."}]}
Also the whole thing needs to be inside {} to make it an object what you had is just an array