How to get a particular field from JSON in Ruby - json

I need to implement a simple shell utility in Ruby which parses JSON from a file and return a particular field from it.
JSON examples to be parsed:
{"status": "fail", "messages": ["Out of capacity"]}
{"status": "success", "messages": [], "result": {"node": {"ip": "1.2.3.4", "description": "", "id": 974, "name": "VM#3"}}}
Idea is to create a CLI utility with two parameters: JSON file to read and field from JSON to extract:
./get_json_field.rb ~/tmp.XXXXXX 'result.node.ip'
./get_json_field.rb ~/tmp.XXXXXX 'messages.0'
I'm struggling how to map 2nd parameter to parsed JSON data structure in Ruby. I can write an iterator for sure, splitting string to an array using dot as separator an go through it item by item but this doesn't look like elegant solution.
Any suggestions for more elegant way?

There is nothing wrong with splitting string and going through parts of it:
require 'json'
data1 = JSON.load('{"status": "fail", "messages": ["Out of capacity"]}')
data2 = JSON.load('{"status": "success", "messages": [], "result": {"node": {"ip": "1.2.3.4", "description": "", "id": 974, "name": "VM#3"}}}')
def get_from_json(data, query)
query.split('.').inject(data) do |memo, key|
key = key.to_i if memo.is_a? Array
memo.fetch(key)
end
end
get_from_json(data1, 'messages.0') # => "Out of capacity"
get_from_json(data2, 'result.node.ip') # => "1.2.3.4"

Take a look at jq it might already do what you are looking for.
jq .messages[0]
jq .node.message.ip
See http://stedolan.github.com/jq/

Related

Need to relationalise a nested JSON string using Pyspark

I am new to Pyspark and need guidance to perform the following task.
A sample data in the form of a JSON string has been given,
{
"id": "1234",
"location": "znd",
"contact": "{\"phone\": [{\"number\":\"12345\",\"code\":\"111\",\"altno\":\"No\"},{\"number\":\"55656\",\"code\":\"222\",\"altno\":\"Yes\"}]}"
}
This needs to be rationalized as follows, as seen below one row of input will get translated to 2 rows.
{id: "1234", "location": "znd","number": "12345", "code": "111","altno":"No"}
{id: "1234", "location": "znd","number": "55656", "code": "222","altno":"No"}
I have tried to use the explode function but as this is a JSON string, explode does not work on it.
I have read the data into a DF and tried to enforce a struct type to later use explode, but that does not work either.

printing json values python

This is my json result from the request
{
"name": "North America",
"region_tag": "na1",
"hostname": "prod.na1.lol.riotgames.com",
"services": [
{
"status": "online",
"incidents": [
{
"active": true,
"created_at": "2018-03-22T10:53:03.397Z",
"id": 7636
}
]
}
]
}
I want to print out the value of status from services
My code
link = "https://na1.api.riotgames.com/lol/status/v3/shard-data?api_key={}".format(API_KEY)
rq_link=rq.get(link).text
rq_json=json.loads(rq_link)
I've tried
print(rq_json['services']['status])
and I get an error saying TypeError: list indices must be integers or slices, not str
Is there another way to do it other than using the for loop
for post in rq_json['services']:
print(rq_json['status'])
The answer you're looking for its
rq_json['services'][0]['status']
Because, the service part is described in a list, you need to specify an index to get into inner dict
You can use list comprehension:
[print(service['status']) for service in rq_json['services']]
More info about list comprehension could be found in the official Python documentation

string to json format powershell

Hey guys I have a singular string output which I need to convert in JSON:
Policy Name: Default_US1 Id: abc123 Buckets: bucket1,bothplaces
Policy Name: Default_CH1 Id: def456 Buckets: support,ch1,ch2
Policy Name: Default_NY2 Id: ghi789 Buckets: demo,bucket1,test1,test
How SHOULD it look like in JSON format?
[
{"Policy Name": "Default_US1"}, {"Id": "abc123"}, {"Buckets":[ "bucket1","bothplaces"]}
{"Policy Name": "Default_CH1"}, {"Id": "def456"}, {"Buckets":[ "support","ch1","ch2"]}
{"Policy Name": "Default_NY2"}, {"Id": "ghi789"}, {"Buckets":[ "demo","bucket1","test1","test"]}
]
above is my current attempt... but other than not working.. I know instinctively it's missing something/s... but I cant figure out what and how to remedy it
Directions on how to do it in Powershell would be a plus, but not necessary
I keep trying but messing up, since I know the best test is making Convertfrom-json show me normal output.
I do not care much how it ends up looking at the end, I just wish to extract all that data, with JSON being the format of choice, any VALID JSON result I can work with and manipulate....but first i need a valid JSON conversion
Ok, so you were correct - your current JSON format is ghastly! The mistake you are making is treating each little bit of data as a separate object when there appears to be a natural hierarchy in your data model.
The following structure more naturally fits your data model. However, this is purely based on a cursory examination of the input data you have posted - I know nothing about the data model itself.
[
{
"Name": "Default_US1",
"Id": "abc123",
"Buckets": [
"bucket1",
"bothplaces"
]
},
{
"Name": "Default_CH1",
"Id": "def456",
"Buckets": [
"support",
"ch1",
"ch2"
]
},
{
"Name": "Default_NY2",
"Id": "ghi789",
"Buckets": [
"demo",
"bucket1",
"test1",
"test2"
]
}
]

Bash Parse JSON Objects jsawk

I´m trying to parse some JSON, which is the output of the Philips Hue API.
I found the tool jsawk, but somehow I´m not able to access the data inside the JSON Object.
The command:
... | jsawk 'return this.f709f9c0b-on-0.name'
works but unfortunately I don't have the ID of each object (e.g. f709f9c0b-on-0). Is there a way to access the object without knowing the ID and then to find out that ID. I tried to use "*" or Iterations of the objects but nothing was working.
Thanks in advance
The output looks like this:
{
"f709f9c0b-on-0": {
"name": "Badezimmer on 0",
"lights": [
"4"
],
"owner": "3e281978544fb15b42bc0e3a3f4ce3",
"recycle": true,
"locked": false,
"appdata": {},
"picture": "",
"lastupdated": "2016-02-17T17:20:06",
"version": 1
},
"69d313be0-on-0": {
"name": "Klavier on 0",
"lights": [
"1"
],
"owner": "3e281978544fb15b42bc0e3a3f4ce3",
"recycle": true,
"locked": false,
"appdata": {},
"picture": "",
"lastupdated": "2016-02-17T17:31:05",
"version": 1
},
...
}
f709f9c0b-on-0 is not a valid identifier due to the hyphens, so you can't use the dot notation. This might work (untested)
... | jsawk 'return this["f709f9c0b-on-0"].name'
I don't have jsawk, but jq can do it like this:
... | jq '.["f709f9c0b-on-0"].name'
Just for the rest of the world.
I solved the problem, by creating a .jar, which handles the problem. I find it much easier to do this in Java, than in bash.
I used this JSON-classes: https://github.com/stleary/JSON-java
Just download the files, create the package org.json and your good to go.
The Java Code, which worked for me is:
String JSON = "your JSON";
JSONObject jsonObject = new JSONObject(JSON);
ArrayList<ArrayList<String>> keyArray = new ArrayList<>();
Iterator<String> keys = jsonObject.keys(); //get all JSON keys
while (keys.hasNext()) { //for all keys do...
String key = (String) keys.next(); //get Current Key
//Now you can access the Object in the Object with:
jsonObject.getJSONObject(key).getString("name")
}
I hope this helps someone.

Unable to loop through JSON output from webservice Python

I have a web-service call (HTTP Get) that my Python script makes in which returns a JSON response. The response looks to be a list of Dictionaries. The script's purpose is to iterate through the each dictionary, extract each piece of metadata (i.e. "ClosePrice": "57.74",) and write each dictionary to its own row in Mssql.
The issue is, I don't think Python is recognizing the JSON output from the API call as a list of dictionaries, and when I try a for loop, I'm getting the error must be int not str. I have tried converting the output to a list, dictionary, tuple. I've also tried to make it work with List Comprehension, with no luck. Further, if I copy/paste the data from the API call and assign it to a variable, it recognizes that its a list of dictionaries without issue. Any help would be appreciated. I'm using Python 2.7.
Here is the actual http call being made: http://test.kingegi.com/Api/QuerySystem/GetvalidatedForecasts?user=kingegi&market=us&startdate=08/19/13&enddate=09/12/13
Here is an abbreviated JSON output from the API call:
[
{
"Id": "521d992cb031e30afcb45c6c",
"User": "kingegi",
"Symbol": "psx",
"Company": "phillips 66",
"MarketCap": "34.89B",
"MCapCategory": "large",
"Sector": "basic materials",
"Movement": "up",
"TimeOfDay": "close",
"PredictionDate": "2013-08-29T00:00:00Z",
"Percentage": ".2-.9%",
"Latency": 37.48089483333333,
"PickPosition": 2,
"CurrentPrice": "57.10",
"ClosePrice": "57.74",
"HighPrice": null,
"LowPrice": null,
"Correct": "FALSE",
"GainedPercentage": 0,
"TimeStamp": "2013-08-28T02:31:08 778",
"ResponseMsg": "",
"Exchange": "NYSE "
},
{
"Id": "521d992db031e30afcb45c71",
"User": "kingegi",
"Symbol": "psx",
"Company": "phillips 66",
"MarketCap": "34.89B",
"MCapCategory": "large",
"Sector": "basic materials",
"Movement": "down",
"TimeOfDay": "close",
"PredictionDate": "2013-08-29T00:00:00Z",
"Percentage": "16-30%",
"Latency": 37.4807215,
"PickPosition": 1,
"CurrentPrice": "57.10",
"ClosePrice": "57.74",
"HighPrice": null,
"LowPrice": null,
"Correct": "FALSE",
"GainedPercentage": 0,
"TimeStamp": "2013-08-28T02:31:09 402",
"ResponseMsg": "",
"Exchange": "NYSE "
}
]
Small Part of code being used:
import os,sys
import subprocess
import glob
from os import path
import urllib2
import json
import time
try:
data = urllib2.urlopen('http://api.kingegi.com/Api/QuerySystem/GetvalidatedForecasts?user=kingegi&market=us&startdate=08/10/13&enddate=09/12/13').read()
except urllib2.HTTPError, e:
print "HTTP error: %d" % e.code
except urllib2.URLError, e:
print "Network error: %s" % e.reason.args[1]
list_id=[x['Id'] for x in data] #test to see if it extracts the ID from each Dict
print(data) #Json output
print(len(data)) #should retrieve the number of dict in list
UPDATE
Answered my own question, here is the method below:
`url = 'some url that is a list of dictionaries' #GetCall
u = urllib.urlopen(url) # u is a file-like object
data = u.read()
newdata = json.loads(data)
print(type(newdata)) # printed data type will show as a list
print(len(newdata)) #the length of the list
newdict = newdata[1] # each element in the list is a dict
print(type(newdict)) # this element is a dict
length = len(newdata) # how many elements in the list
for a in range(1,length): #a is a variable that increments itself from 1 until a number
var = (newdata[a])
print(var['Correct'], var['User'])`