Extract specific info from json-list - json

In my system from json-call http://192.168.1.6:8080/json.htm?type=devices&rid=89 I get the output below.
{
"ActTime" : 1501360852,
"ServerTime" : "2017-07-29 22:40:52",
"Sunrise" : "05:50",
"Sunset" : "21:28",
"result" : [
{
"AddjMulti" : 1.0,
"AddjMulti2" : 1.0,
"AddjValue" : 0.0,
"AddjValue2" : 0.0,
"BatteryLevel" : 255,
"CustomImage" : 0,
"Data" : "73 Lux",
"Description" : "",
"Favorite" : 1,
"HardwareID" : 4,
"HardwareName" : "Dummies",
"HardwareType" : "Dummy (Does nothing, use for virtual switches only)",
"HardwareTypeVal" : 15,
"HaveTimeout" : true,
"ID" : "82089",
"LastUpdate" : "2017-07-29 21:16:22",
"Name" : "ESP8266C_Licht1",
"Notifications" : "false",
"PlanID" : "0",
"PlanIDs" : [ 0 ],
"Protected" : false,
"ShowNotifications" : true,
"SignalLevel" : "-",
"SubType" : "Lux",
"Timers" : "false",
"Type" : "Lux",
"TypeImg" : "lux",
"Unit" : 1,
"Used" : 1,
"XOffset" : "0",
"YOffset" : "0",
"idx" : "89"
}
],
"status" : "OK",
"title" : "Devices"
}
'Automating' such call by the below scripttime lua-script is aimed at extraction of specific information, to be further used in applications.
The first 11 lines run without problems, but further extraction of the information is a problem.
I have tried various scriptlines to get a solution for A), B), C), D) and E), but either they generate error-report, or they don't give results: see further below for the 'best' trial-scriptlines and related results.
To avoid misunderstanding: those dashed commentlines in the script just below this question with A), B), C), D) and E) are only describing the desired actions/functions and are in no way meant as scriptlines!
Question:
Help requested in the form of better applicable scriptlines for A) till E) in the trialscript at the end of this message, or hints where to find applicable example scriptlines.
-- Lua-script to determine staleness, time-out and value for data from Json-call
print('Start of Timeout-script')
commandArray = {}
TimeOutLimit = 10 -- Allowed timeout in seconds
json = (loadfile "/home/pi/domoticz/scripts/lua/JSON.lua")() -- For Linux
-- json = (loadfile "D:\\Domoticz\\scripts\\lua\\json.lua")() -- For Windows
-- Line 07
local content=assert(io.popen('curl "http://192.168.1.6:8080/json.htm?type=devices&rid=89"')) -- notice double quotes
local list = content:read('*all')
content:close()
local jsonList = json:decode(list)
-- Line 12 Next scriptlines describe desired actions
-- A) Extract ServerTime as numeric value (not as string)
-- B) Extract LastUpdate as numeric value (not as string)
-- Staleness = ServerTime - LastUpdate
-- C) Extract HaveTimeout as boolean (not as string)
-- If HaveTimeout and (Staleness > TimeOutlimit) then
-- Print('TimeOutLimit exceeded by ' .. (Staleness - TimeOutLimit) .. 'seconds')
-- End
-- D) Extract textstring from Type or Data
-- E) Extract numeric value from Data
print('End of Timeout-script')
return commandArray
For lines 11 etc, the following trial-scriptlines gave 'best' results (= no errors):
-- Line 11
-- local Servertime = json:decode(ServerTime)
-- print('Servertime : '..Servertime)
-- Line 14
-- CheckTimeOut =jsonValue.result[1].HaveTimeout -- value from "HaveTimeout", inside "result" bloc number 1 (even if it's the only one)
CurrentServerTime =jsonValue.Servertime -- value from "ServerTime"
CurrentLastUpdate = jsonValue.result[1].LastUpdate
CurrentData = jsonValue.result[1].Data
-- Line 19
print('TimeOut : '..CheckTimeOut)
print('Servertime : '..CurrentServerTime)
print('LastUpdate : '..CurrentLastUpdate)
print('Data-content : '..CurrentData)
print('End of Timeout-script')
return commandArray
Results:
Without dashes before the lines 12 and 13, respectively 15, then the following error reports:
660: nil passed to JSON:decode()
lua:15: attempt to index global 'jsonValue' (a nil value)
With dashes before lines 12, 13 and 15 for the trial-scriptlines shown above, according to the log no errors exist (as demonstrated by the 2 prints)
2017-07-31 16:30:02.520 LUA: Start of Timeout-script
2017-07-31 16:30:02.563 LUA: End of Timeout-script
But why no print-results in the log from Lines 20 till 23?
Not having those print-results makes it difficult to determine next steps in data-extraction, to achieve the objectives described under A) till E).
;-) Error reports generally are more useful information than "no errors, but no results"

Without downloading content over the network, the working script looks like this:
local json = require("json")
local jj = [[
{
"ActTime" : 1501360852,
"ServerTime" : "2017-07-29 22:40:52",
"Sunrise" : "05:50",
"Sunset" : "21:28",
"result" : [
{
"AddjMulti" : 1.0,
"AddjMulti2" : 1.0,
"AddjValue" : 0.0,
"AddjValue2" : 0.0,
"BatteryLevel" : 255,
"CustomImage" : 0,
"Data" : "73 Lux",
"Description" : "",
"Favorite" : 1,
"HardwareID" : 4,
"HardwareName" : "Dummies",
"HardwareType" : "Dummy (Does nothing, use for virtual switches only)",
"HardwareTypeVal" : 15,
"HaveTimeout" : true,
"ID" : "82089",
"LastUpdate" : "2017-07-29 21:16:22",
"Name" : "ESP8266C_Licht1",
"Notifications" : "false",
"PlanID" : "0",
"PlanIDs" : [ 0 ],
"Protected" : false,
"ShowNotifications" : true,
"SignalLevel" : "-",
"SubType" : "Lux",
"Timers" : "false",
"Type" : "Lux",
"TypeImg" : "lux",
"Unit" : 1,
"Used" : 1,
"XOffset" : "0",
"YOffset" : "0",
"idx" : "89"
}
],
"status" : "OK",
"title" : "Devices"
}
]]
print('Start of Timeout-script')
local jsonValue = json.decode(jj)
CheckTimeOut =jsonValue.result[1].HaveTimeout
CurrentServerTime =jsonValue.Servertime
CurrentLastUpdate = jsonValue.result[1].LastUpdate
CurrentData = jsonValue.result[1].Data
print('TimeOut : '.. (CheckTimeOut and "true" or "false") )
print('Servertime : '.. (CurrentServerTime or "nil") )
print('LastUpdate : '.. (CurrentLastUpdate or "nil") )
print('Data-content : '.. (CurrentData or "nil") )
print('End of Timeout-script')
result:
Start of Timeout-script
TimeOut : true
Servertime : nil
LastUpdate : 2017-07-29 21:16:22
Data-content : 73 Lux
End of Timeout-script

Related

JSON - Character issue

I extracted some data from a mongo database using the RMongo library. I have been working with the data with no problem. However, I need to access a field that was saved, originally in the database, as JSON. Since rmongodb saves the data as data frame, I now have a large character vector of length 1:
res1 = "[ { \"text\" : \"#Kayture Beyoncé jam session ?\" , \"name\" : \"beponcé \xed\xa0\xbc\xed\xbc\xbb\" , \"screenName\" : \"ColaaaaTweedy\" , \"follower\" : false , \"mentions\" : [ \"Kayture\"] , \"userTwitterId\" : \"108061963\"} , { \"text\" : \"#Kayture fucking marry me\" , \"name\" : \"George McQueen\" , \"screenName\" : \"GeorgeMcQueen12\" , \"follower\" : false , \"mentions\" : [ \"Kayture\"] , \"userTwitterId\" : \"67896750\"}]"
I need to extract all the "text" attributes of the objects from this array (there are 2 in this example), but I can not figure out a fast way. I was trying using strsplit, or going from character to json files using jsonlite, and then to list, but it does not work.
Any ideas?
Thanks!
Starting from
res1 = "[ { \"text\" : \"#Kayture Beyoncé jam session ?\" , \"name\" : \"beponcé \xed\xa0\xbc\xed\xbc\xbb\" , \"screenName\" : \"ColaaaaTweedy\" , \"follower\" : false , \"mentions\" : [ \"Kayture\"] , \"userTwitterId\" : \"108061963\"} , { \"text\" : \"#Kayture fucking marry me\" , \"name\" : \"George McQueen\" , \"screenName\" : \"GeorgeMcQueen12\" , \"follower\" : false , \"mentions\" : [ \"Kayture\"] , \"userTwitterId\" : \"67896750\"}]"
you can use fromJSON() from the jsonlite package to parse that JSON object.
library(jsonlite)
fromJSON(res1)
text name screenName follower mentions userTwitterId
1 #Kayture Beyoncé jam session ? beponcé í ¼í¼» ColaaaaTweedy FALSE Kayture 108061963
2 #Kayture fucking marry me George McQueen GeorgeMcQueen12 FALSE Kayture 67896750

How to Parse Json on Scala

Here, I attached my code (pure Scala)
package stock
import scala.io.Source._
import java.util.Scanner
import java.io.File
import scala.util.parsing.json._
object StockDetails {
def main(args : Array[String]){
val filePath = new java.io.File(".").getCanonicalPath
val source = scala.io.Source.fromFile(filePath +"/stock/file.txt")
// file.txt --> contains
// line 1 --> GOOG - 50, MS - 10
// line 2 --> SGI - 100, GOOG - 50, MS - 10
// line 3 --> GOOG - 100, AMZN - 90, MS - 80
for(line <- source.getLines()) {
val txt = line.split(",")
val s1 = txt.map{ss =>
val s2 = ss.split(" - ")(0).trim()
val holder = fromURL("http://finance.google.com/finance/info?client=ig&q="+s2).mkString
println("holder===========",holder)
val s3 = holder.split("//")(1)
println("s3================",s3)
val s4 = JSON.parseFull(s3).get
println("s4==========================",s4)
}
}
}
}
O\P:
(holder==========================,
// [
{
"id": "660479"
,"t" : "MS"
,"e" : "NYSE"
,"l" : "43.06"
,"l_fix" : "43.06"
,"l_cur" : "43.06"
,"s": "0"
,"ltt":"4:02PM EST"
,"lt" : "Dec 23, 4:02PM EST"
,"lt_dts" : "2016-12-23T16:02:12Z"
,"c" : "+0.27"
,"c_fix" : "0.27"
,"cp" : "0.63"
,"cp_fix" : "0.63"
,"ccol" : "chg"
,"pcls_fix" : "42.79"
}
])
(s3================,
[{
"id": "660479"
,"t" : "MS"
,"e" : "NYSE"
,"l" : "43.06"
,"l_fix" : "43.06"
,"l_cur" : "43.06"
,"s": "0"
,"ltt":"4:02PM EST"
,"lt" : "Dec 23, 4:02PM EST"
,"lt_dts" : "2016-12-23T16:02:12Z"
,"c" : "+0.27"
,"c_fix" : "0.27"
,"cp" : "0.63"
,"cp_fix" : "0.63"
,"ccol" : "chg"
,"pcls_fix" : "42.79"
}
])
(s4==========================,
List(Map(
e -> NYSE,
s -> 0,
cp_fix -> 0.63,
l_cur -> 43.06,
ccol -> chg,
t -> MS,
pcls_fix -> 42.79,
id -> 660479,
l -> 43.06,
l_fix -> 43.06,
c_fix -> 0.27,
c -> +0.27,
cp -> 0.63,
lt -> Dec 23, 4:02PM EST,
lt_dts -> 2016-12-23T16:02:12Z,
ltt -> 4:02PM EST)))
Here, I want the " l " value but i can't able to get it, When I used map/ foreach it's returned as
$ scalac Stock.scala
Stock.scala:30: error: value map is not a member of Any
val s5 = s4.map{ ex => ex }
^
one error found
And tried this link but i can't able to get it, Here what i do?
Parsing using JSON.parseFull returns option of any.
scala> JSON.parseFull("""{"key": "value"}""")
res2: Option[Any] = Some(Map(key -> value))
So, use pattern matching to extract the types you want.
or type cast (not recommended)
Well... the thing is that you have S4 as Any.
// since we know that s4 is supposed to be a List
// we can type-cast it to List[Any]
val s4List = s4.asInstanceOf[List[Any]]
// now we also know that s4 was actually a list of Map's
// and those maps look to be String -> String
// so we can type-cast all the things inside list to Map[String, String]
val s4MapList = s4List.map(m => m.asInstanceOf[Map[String, String]])
// and we also know that s4 is a List of Map's and has length 1
val map = s4MapList(0)
// lets say you wanted id
val idOption = map.get("id")
// but we know that map has a key "id"
// so we can just get that
val id = map("id")
Note :: This is not the recommended way of doing things in Scala, but can help you understand what are you supposed to do on a basic level. Keep learning... and you will come to understand better ways to deal with things.

Loading JSON data to a list in a particular order using PyMongo

Let's say I have the following document in a MongoDB database:
{
"assist_leaders" : {
"Steve Nash" : {
"team" : "Phoenix Suns",
"position" : "PG",
"draft_data" : {
"class" : 1996,
"pick" : 15,
"selected_by" : "Phoenix Suns",
"college" : "Santa Clara"
}
},
"LeBron James" : {
"team" : "Cleveland Cavaliers",
"position" : "SF",
"draft_data" : {
"class" : 2003,
"pick" : 1,
"selected_by" : "Cleveland Cavaliers",
"college" : "None"
}
},
}
}
I'm trying to collect a few values under "draft_data" for each player in an ORDERED list. The list needs to look like the following for this particular document:
[ [1996, 15, "Phoenix Suns"], [2003, 1, "Cleveland Cavaliers"] ]
That is, each nested list must contain the values corresponding to the "pick", "selected_by", and "class" keys, in that order. I also need the "Steve Nash" data to come before the "LeBron James" data.
How can I achieve this using pymongo? Note that the structure of the data is not set in stone so I can change this if that makes the code simpler.
I'd extract the data and turn it into a list in Python, once you've retrieved the document from MongoDB:
for doc in db.collection.find():
for name, info in doc['assist_leaders'].items():
draft_data = info['draft_data']
lst = [draft_data['class'], draft_data['pick'], draft_data['selected_by']]
print name, lst
List comprehension is the way to go here (Note: don't forget .iteritems() in Python2 or .items() in Python3 or you'll get a ValueError: too many values to unpack).
import pymongo
import numpy as np
client = pymongo.MongoClient()
db = client[database_name]
dataList = [v for i in ["Steve Nash", "LeBron James"]
for key in ["class", "pick", "selected_by"]
for document in db.collection_name.find({"assist_leaders": {"$exists": 1}})
for k, v in document["assist_leaders"][i]["draft_data"].iteritems()
if k == key]
print dataList
# [1996, 15, "Phoenix Suns", 2003, 1, "Cleveland Cavaliers"]
matrix = np.reshape(dataList, [2,3])
print matrix
# [ [1996, 15, "Phoenix Suns"],
# [2003, 1, "Cleveland Cavaliers"] ]

Parsing JSON from Google Distance Matrix API with Corona SDK

So I'm trying to pull data from a JSON string (as seen below). When I decode the JSON using the code below, and then attempt to index the duration text, I get a nil return. I have tried everything and nothing seems to work.
Here is the Google Distance Matrix API JSON:
{
"destination_addresses" : [ "San Francisco, CA, USA" ],
"origin_addresses" : [ "Seattle, WA, USA" ],
"rows" : [
{
"elements" : [
{
"distance" : {
"text" : "1,299 km",
"value" : 1299026
},
"duration" : {
"text" : "12 hours 18 mins",
"value" : 44303
},
"status" : "OK"
}]
}],
"status" : "OK"
}
And here is my code:
local json = require ("json")
local http = require("socket.http")
local myNewData1 = {}
local SaveData1 = function (event)
distanceReturn = ""
distance = ""
local URL1 = "http://maps.googleapis.com/maps/api/distancematrix/json?origins=Seattle&destinations=San+Francisco&mode=driving&&sensor=false"
local response1 = http.request(URL1)
local data2 = json.decode(response1)
if response1 == nil then
native.showAlert( "Data is nill", { "OK"})
print("Error1")
distanceReturn = "Error1"
elseif data2 == nill then
distanceReturn = "Error2"
native.showAlert( "Data is nill", { "OK"})
print("Error2")
else
for i = 1, #data2 do
print("Working")
print(data2[i].rows)
for j = 1, #data2[i].rows, 1 do
print("\t" .. data2[i].rows[j])
for k = 1, #data2[i].rows[k].elements, 1 do
print("\t" .. data2[i].rows[j].elements[k])
for g = 1, #data2[i].rows[k].elements[k].duration, 1 do
print("\t" .. data2[i].rows[k].elements[k].duration[g])
for f = 1, #data2[i].rows[k].elements[k].duration[g].text, 1 do
print("\t" .. data2[i].rows[k].elements[k].duration[g].text)
distance = data2[i].rows[k].elements[k].duration[g].text
distanceReturn = data2[i].rows[k].elements[k].duration[g].text
end
end
end
end
end
end
timer.performWithDelay (100, SaveData1, 999999)
Your loops are not correct. Try this shorter solution.
Replace all your "for i = 1, #data2 do" loop for this one below:
print("Working")
for i,row in ipairs(data2.rows) do
for j,element in ipairs(row.elements) do
print(element.duration.text)
end
end
This question was solved on Corona Forums by Rob Miracle (http://forums.coronalabs.com/topic/47319-parsing-json-from-google-distance-matrix-api/?hl=print_r#entry244400). The solution is simple:
"JSON and Lua tables are almost identical data structures. In this case your table data2 has top level entries:
data2.destination_addresses
data2.origin_addresses
data2.rows
data2.status
Now data2.rows is another table that is indexed by numbers (the [] brackets) but here is only one of them, but its still an array entry:
data.rows[1]
Then inside of it is another numerically indexed table called elements.
So far to get to the element they are (again there is only one of them
data2.rows[1].elements[1]
then it's just accessing the remaining elements:
data2.rows[1].elements[1].distance.text
data2.rows[1].elements[1].distance.value
data2.rows[1].elements[1].duration.text
data2.rows[1].elements[1].duration.value
There is a great table printing function called print_r which can be found in the community code which is great for dumping tables like this to see their structure."

Problems reading JSON file in R

I have a JSON file (an export from mongoDB) that I'd like to load into R. The document is about 890 MB in size with roughly 63,000 rows of 12 fields. The fields are numeric, character and date. I'd like to end up with a 63000 x 12 data frame.
lines <- readLines("fb2013.json")
result: jFile has all 63,000 elements in char class and all fields are lumped into one field.
Each file looks something like this:
"{ \"_id\" : \"10151271769737669\", \"comments_count\" : 36, \"created_at\" : { \"$date\" : 1357941938000 }, \"icon\" : \"http://blahblah.gif\", \"likes_count\" : 450, \"link\" : \"http://www.blahblahblah.php\", \"message\" : \"I wish I could figure this out!\", \"page_category\" : \"Computers\", \"page_id\" : \"30968999999\", \"page_name\" : \"NothingButTrouble\", \"type\" : \"photo\", \"updated_at\" : { \"$date\" : 1358210153000 } }"
Using rjson,
jFile <- fromJSON(paste(readLines("fb2013.json"), collapse=""))
only the first row is read into jFile but there are 12 fields.
Using RJSONIO:
jFile <- fromJSON(lines)
results in the following:
Warning messages:
1: In if (is.na(encoding)) return(0L) :
the condition has length > 1 and only the first element will be used
Again, only the first row is read into jFile and there are 12 fields.
The output from rjson and RJSONIO looks something like this:
$`_id`
[1] "1018535"
$comments_count
[1] 0
$created_at
$date
1.357027e+12
$icon
[1] "http://blah.gif"
$likes_count
[1] 20
$link
[1] "http://www.chachacha"
$message
[1] "I'd love to figure this out."
$page_category
[1] "Internet/software"
$page_id
[1] "3924395872345878534"
$page_name
[1] "Not Entirely Hopeless"
$type
[1] "photo"
$updated_at
$date
1.357027e+12
try
library(rjson)
path <- "WHERE/YOUR/JSON/IS/SAVED"
c <- file(path, "r")
l <- readLines(c, -1L)
json <- lapply(X=l, fromJSON)
Since you want a data.frame, try this:
# three copies of your sample...
line.1<- "{ \"_id\" : \"10151271769737669\", \"comments_count\" : 36, \"created_at\" : { \"$date\" : 1357941938000 }, \"icon\" : \"http://blahblah.gif\", \"likes_count\" : 450, \"link\" : \"http://www.blahblahblah.php\", \"message\" : \"I wish I could figure this out!\", \"page_category\" : \"Computers\", \"page_id\" : \"30968999999\", \"page_name\" : \"NothingButTrouble\", \"type\" : \"photo\", \"updated_at\" : { \"$date\" : 1358210153000 } }"
line.2<- "{ \"_id\" : \"10151271769737669\", \"comments_count\" : 36, \"created_at\" : { \"$date\" : 1357941938000 }, \"icon\" : \"http://blahblah.gif\", \"likes_count\" : 450, \"link\" : \"http://www.blahblahblah.php\", \"message\" : \"I wish I could figure this out!\", \"page_category\" : \"Computers\", \"page_id\" : \"30968999999\", \"page_name\" : \"NothingButTrouble\", \"type\" : \"photo\", \"updated_at\" : { \"$date\" : 1358210153000 } }"
line.3<- "{ \"_id\" : \"10151271769737669\", \"comments_count\" : 36, \"created_at\" : { \"$date\" : 1357941938000 }, \"icon\" : \"http://blahblah.gif\", \"likes_count\" : 450, \"link\" : \"http://www.blahblahblah.php\", \"message\" : \"I wish I could figure this out!\", \"page_category\" : \"Computers\", \"page_id\" : \"30968999999\", \"page_name\" : \"NothingButTrouble\", \"type\" : \"photo\", \"updated_at\" : { \"$date\" : 1358210153000 } }"
x <- paste(line.1, line.2, line.3, sep="\n")
lines <- readLines(textConnection(x))
library(rjson)
# this is the important bit
df <- data.frame(do.call(rbind,lapply(lines,fromJSON)))
ncol(df)
# [1] 12
# finally, there's some cleaning up to do...
df$created_at
# [[1]]
# [[1]]$`$date`
# [1] 1.357942e+12
# ...
df$created_at <- as.POSIXlt(unname(unlist(df$created_at)/1000),origin="1970-01-01")
df$created_at
# [1] "2013-01-11 17:05:38 EST" "2013-01-11 17:05:38 EST" "2013-01-11 17:05:38 EST"
df$updated_at <- as.POSIXlt(unname(unlist(df$updated_at)/1000),origin="1970-01-01")
Note that this conversion assumes that the dates were stored as milliseconds since the epoch.