How is it possible to get the input of an json file:
{ "name": "John",
"work": "chef",
"age": "29",
"messages": [
{
"msg_name": "Hello",
"msg": "how_are_you"
},
{ "second_msg_name": "hi",
"msg": "fine"
}
]
}
into a Lua table? All json.lua scripts I found did not work for JSON with new lines. Does someone know a solution?
So piglets solution works for the string in the same script.
But how does it work with a JSON file?
local json = require("dkjson")
local file = io.open("C:\\Users\\...\\Documents\\Lua_Plugins\\test_file_reader\\test.json", "r")
local myTable = json.decode(file)
print(myTable)
Here then comes the error "bad argument #1 to 'strfind' (string expected, got FILE*)" on. Does someone see my fault?
local json = require("dkjson")
local yourString = [[{ "name": "John",
"work": "chef",
"age": "29",
"messages": [
{
"msg_name": "Hello",
"msg": "how_are_you"
},
{ "second_msg_name": "hi",
"msg": "fine"
}
]
}]]
local myTable = json.decode(yourString)
http://dkolf.de/src/dkjson-lua.fsl/home
I found the solution:
local json = require("dkjson")
local file = io.open("C:\\Users\\...\\Documents\\Lua_Plugins\\test_file_reader\\test.json", "r")
local content = file:read "*a"
file:close()
local myTable = json.decode(content)
Related
I am developing a telegram bot with python (telebot) , aws lambda and api gateway.
I have a problem in the lambda function and I can't understand why I have this kind of problem.
My lambda is this:
import telebot
import datetime
TOKEN = 'xxx'
def lambda_handler(event, context):
bot = telebot.TeleBot(TOKEN)
# Extract the message key over payload's body
message = json.loads(event['body'])
print(message)
# Split between three variables bellow
chat_id = message['chat']['id'] # Chat ID will guide your chatbot reply
sender = message['from']['first_name'] # Sender's first name, registered by user's telegram app
text = message['text'] # The message content
if text.lower().strip() == "/time":
current_time = datetime.strftime(datetime.now(), "%H:%M:%S")
bot.send_message(chat_id, "Right now its {} UTC.".format(current_time))
else:
pass
The error I get, running the test is this:
Response
{
"errorMessage": "'body'",
"errorType": "KeyError",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 10, in lambda_handler\n message = json.loads(event['body'])\n"
]
}
The given json file:
{
"update_id": 0000000,
"message": {
"message_id": 000000,
"from": {
"id": 00000000,
"is_bot": false,
"first_name": "myname",
"last_name": "mysurname",
"username": "sursurname",
"language_code": "it"
},
"chat": {
"id": 000000,
"first_name": "myname",
"last_name": "mysurname",
"username": "sursurname",
"type": "private"
},
"date": 1654697178,
"forward_from": {
"id": 00000000,
"is_bot": false,
"first_name": "mysurname",
"last_name": "mysurname",
"username": "sursurname",
"language_code": "it"
},
"forward_date": 0000000000,
"text": "ciao"
}
}
I cannot understand why it is not able to read the body in any way, maybe I am in the wrong library? Do you have any suggestions to help me with this?
event['body'] is almost definitely not the correct key to access data passed through an event. The event passes information in a nested dictionary and you'll need to figure out how to drill down to the correct key.
I solved it by doing this:
json_str = json.dumps(event)
resp = json.loads(json_str)
chat_id = resp['message']['chat']['id']
message_text = resp['message']['text']
message_chat_id = resp['message']['chat']['id']
message_username = resp['message']['from']['first_name']
bot = telebot.TeleBot(TOKEN)
bot.send_message(chat_id, "Hey, I understood this message!, hi {}".format(message_username))
I can't access my JSON model which has been defined in the manifest.json.
My JSON data "zCatsTestJ" looks like this:
{
"d": {
"results": [{
"status": "30",
"skostl": "6210",
"catshours": "2.50",
"ktext": "-",
"counter": "000003484040",
"mandt": "101",
"pernr": "00015822",
"usrid": "-",
"workdate": "\/Date(1477267200000)\/",
"raufnr": "6000025",
}, {
"status": "30",
"skostl": "6210",
"catshours": "2.50",
"ktext": "-",
"counter": "000003484040",
"mandt": "101",
"pernr": "00015822",
"usrid": "-",
"workdate": "\/Date(1477267200000)\/",
"raufnr": "6000025",
}]
}
}
The model seems to be accesible as sJsonDate1 is showing me the data in the console but I can't access a single date. In the end I want to loop over those dates and change the formatting.
var sJsonDate1 = this.getOwnerComponent().getModel("zCatsTestJ");
var sJsonDate2 =this.getOwnerComponent().getModel("zCatsTestJ").getProperty("/d/results/1/workdate");
console.log(sJsonDate1);
console.log(sJsonDate2);
Here is the console output where I can see the complete data.
Console sJsonDate1
But when I try to access one datapoint it says undefined
Console sJsonDate2
I have also instatiated the Model directly in the component and it is working fine. When I compare the model object from getOwnerComponent() and the new one they are nearly the same except for the local one having no aBindings
Model comparison in console
Any help will be highly apreciated. Thanks
Your JSON is invalid, status is missing " and you never close the array. I tried it locally after correcting your JSON and it worked fine on my end.
var json = {
"d": {
"results": [
{
"status": "30",
"skostl": "6210",
"catshours": "2.50",
"ktext": "-",
"counter": "000003484040",
"mandt": "101",
"pernr": "00015822",
"usrid": "-",
"workdate": "\/Date(1477267200000)\/",
"raufnr": "6000025",
}, {
"status": "30",
"skostl": "6210",
"catshours": "2.50",
"ktext": "-",
"counter": "000003484040",
"mandt": "101",
"pernr": "00015822",
"usrid": "-",
"workdate": "\/Date(1477267200000)\/",
"raufnr": "6000025",
}]
}
};
var myModel = new JSONModel(json);
this.getView().setModel(myModel, "test");
this.getView().getModel("test").getProperty("/d/results/1/workdate"); "/Date(1477267200000)/"
Here is the solution. Somehow the JsonModel was too big (667 Data Points) and wasn't loaded on the time of calling it. That's why it has worked with only two entries.
myModel.attachRequestCompleted(function() {
var sJsonDate = myModel.getProperty("/d/results/1/workdate");
console.log(sJsonDate);
});
I have fake users.json file and I can http.get to list the array of json.
Since I want to get the particular user by id and haven't stored the data in the database, instead just use the fake json data.
[
{
"id": "cb55524d-1454-4b12-92a8-0437e8e6ede7",
"name": "john",
"age": "25",
"country": "germany"
},
{
"id": "ab55524d-1454-4b12-92a8-0437e8e6ede8",
"name": "tom",
"age": "28",
"country": "canada"
}
]
I can do this stuff if the data is stored in the database, but not sure how to proceed with the fake json data.
Any help is appreciated.
Thanks
If you need the json as raw data, for just fake data, You can simply require it and use it as object..
const JsonObj = require('path/to/file.json')
console.log(JsonObj[0].id) // <-- cb55524d-1454-4b12-92a8-0437e8e6ede7
Plus, if you need more dynamic solution, there is a good JSON-server you can easily use for testing and so: check this git repo
var _ = require('underscore');
var dummyJson = [
{
"id": "cb55524d-1454-4b12-92a8-0437e8e6ede7",
"name": "john",
"age": "25",
"country": "germany"
},
{
"id": "ab55524d-1454-4b12-92a8-0437e8e6ede8",
"name": "tom",
"age": "28",
"country": "canada"
}
]
var requiredID = "cb55524d-1454-4b12-92a8-0437e8e6ede7";
var reuiredObject = _.find(dummyJson, function (d) {
return d.id === requiredID;
})
Get JSON object using JSON.parse('users.json') and store it in a variable users.
Loop through array of users using for .. in and using if condition on id update the object if required.
Stringify the updated users object using JSON.stringify(users); and write this string to users.json file using fs.write() module in NodeJS so you will have updated objects in your json file.
I have a text file which contains raw data. I want to parse that data and clean it so that it can be used further.The following is the rawdata.
"{\x0A \x22identifier\x22: {\x0A \x22company_code\x22: \x22TSC\x22,\x0A \x22product_type\x22: \x22airtime-ctg\x22,\x0A \x22host_type\x22: \x22android\x22\x0A },\x0A \x22id\x22: {\x0A \x22type\x22: \x22guest\x22,\x0A \x22group\x22: \x22guest\x22,\x0A \x22uuid\x22: \x221a0d4d6e-0c00-11e7-a16f-0242ac110002\x22,\x0A \x22device_id\x22: \x22423e49efa4b8b013\x22\x0A },\x0A \x22stats\x22: [\x0A {\x0A \x22timestamp\x22: \x222017-03-22T03:21:11+0000\x22,\x0A \x22software_id\x22: \x22A-ACTG\x22,\x0A \x22action_id\x22: \x22open_app\x22,\x0A \x22values\x22: {\x0A \x22device_id\x22: \x22423e49efa4b8b013\x22,\x0A \x22language\x22: \x22en\x22\x0A }\x0A }\x0A ]\x0A}"
I want to remove all the hexadecimal characters,I tried parsing the data and storing in an array and cleaning it using re.sub() but it gives the same data.
for line in f:
new_data = re.sub(r'[^\x00-\x7f],\x22',r'', line)
data.append(new_data)
\x0A is the hex code for newline. After s = <your json string>, print(s) gives
>>> print(s)
{
"identifier": {
"company_code": "TSC",
"product_type": "airtime-ctg",
"host_type": "android"
},
"id": {
"type": "guest",
"group": "guest",
"uuid": "1a0d4d6e-0c00-11e7-a16f-0242ac110002",
"device_id": "423e49efa4b8b013"
},
"stats": [
{
"timestamp": "2017-03-22T03:21:11+0000",
"software_id": "A-ACTG",
"action_id": "open_app",
"values": {
"device_id": "423e49efa4b8b013",
"language": "en"
}
}
]
}
You should parse this with the json module load (from file) or loads (from string) functions. You will get a dict with 2 dicts and a list with a dict.
EDIT 2014-05-01: I tried fromJSON first (as suggested below), but that only parsed the first line. I found out that there were commas missing between the brackets of each JSON line so I changed that in TextEdit and saved the file. I also added [ at the beginning of the file and ] at the end and then it worked with JSON. Now the next step: from a list (with embedded lists) to a dataframe (or csv).
I get a data package from edX every now and then on the courses we are evaluating. Some of these are just plain .csv files which are quite easy to handle, others are more difficult for me (not having a CS or programming background).
I have 2 files I want to open and parse into csv files for analysis in R. I have tried many many json2csv tools out there, but to no avail. I also tried the simple methods described here to turn json into csv.
The data is confidential, so I cannot share the entire data set, but will share the first two lines of the file, maybe that helps. The problem is that nowhere I find anything about .mongo files, which to me seems quite strange, do they even exist? Or is this just a JSON file that may be corrupted (which could explain the errors)?
Any suggestions are welcome.
The first 2 lines in one of the .mongo files:
{
"_id": {
"$oid": "52d1e62c350e7a3156000009"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
],
"at_position_list": [
],
"body": "the delft university accredited course with the scholarship (fundamentals of water treatment) is supposed to start in about a month's time. But have the scholarship list been published? Any tentative date??",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"author_id": "269835",
"comment_thread_id": {
"$oid": "52cd40c5ab40cf347e00008d"
},
"author_username": "tachak59",
"sk": "52d1e62c350e7a3156000009",
"updated_at": {
"$date": 1389487660636
},
"created_at": {
"$date": 1389487660636
}
}{
"_id": {
"$oid": "52d0a66bcb3eee318d000012"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
{
"$oid": "52c63278100c07c0d1000028"
}
],
"at_position_list": [
],
"body": "I got it. Thank you!",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"parent_id": {
"$oid": "52c63278100c07c0d1000028"
},
"author_id": "2655027",
"comment_thread_id": {
"$oid": "52c4f303b03c4aba51000013"
},
"author_username": "dmoronta",
"sk": "52c63278100c07c0d1000028-52d0a66bcb3eee318d000012",
"updated_at": {
"$date": 1389405803386
},
"created_at": {
"$date": 1389405803386
}
}{
"_id": {
"$oid": "52ceea0cada002b72c000059"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
{
"$oid": "5287e8d5906c42f5aa000013"
}
],
"at_position_list": [
],
"body": "if u please send by mail \n",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"parent_id": {
"$oid": "5287e8d5906c42f5aa000013"
},
"author_id": "2276302",
"comment_thread_id": {
"$oid": "528674d784179607d0000011"
},
"author_username": "totah1993",
"sk": "5287e8d5906c42f5aa000013-52ceea0cada002b72c000059",
"updated_at": {
"$date": 1389292044203
},
"created_at": {
"$date": 1389292044203
}
}
R doesn't have "native" support for these files but there is a JSON parser with the rjson package. So I might load my .mongo file with:
myfile <- "path/to/myfile.mongo"
myJSON <- readLines(myfile)
myNiceData <- fromJSON(myJSON)
Since RJson converts into a data structure that fits the object being read, you'll have to do some additional snooping but once you have an R data type you shouldn't have any trouble working with it from there.
Another package to consider when parsing JSON data is jsonlite. It will make data frames for you so you can write them to a csv format with write.table or some other applicable method for writing objects.
NOTE: if it is easier to connect to the MongoDB and get the data from a request, then RMongo may be a good bet. The R-Bloggers also made a post about using RMongo that has a nice little walkthrough.
I used RJSON as suggested by #theWanderer and with the help of a colleague wrote the following code to parse the data into columns, choosing the specific columns that are needed, and checking each of the instances if they return the right variables.
Entire workflow:
Checked some of the data in jsonlint - corrected the errors → },{ instead of }{ between each line and [ and ] at the beginning and end of the file
Made a smaller file to play with, containing about 11 JSON lines
Used the code below to parse the datafile - however, checking the different listItems first if they are not lists themselves (that gives problems) // as you will see, I also removed things like \n because that gave errors and added an empty value for parent_id if there is none in the data (otherwise it would mix up the data)
The code to import the .mongo file into R and then parse it into CSV:
library(rjson)
###### set working directory to write out the data file
setwd("/your/favourite/dir/json to csv/")
#never ever convert strings to factors
options(stringsAsFactors = FALSE)
#import the .mongo file to R
temp.data = fromJSON(file="temp.mongo", method="C", unexpected.escape="error")
file.remove("temp.csv") ## removes the old datafile if there is one
## (so the data is not appended to the file,
## but a new file is created)
listItem = temp.data[[1]] ## prepare the listItem the first time
for (listItem in temp.data){
parent_id = ""
if (length(listItem$parent_id)>0){
parent_id = listItem$parent_id
}
write.table(t(c(
listItem$votes$up_count, listItem$visible, parent_id,
gsub("\n", "", listItem$body), listItem$course_id, unlist(listItem["_type"]),
listItem$endorsed, listItem$anonymous, listItem$author_id,
unlist(listItem$comment_thread_id), listItem$author_username,
as.POSIXct(unlist(listItem$created_at)/1000, origin="1970-01-01"))), # end t(), c()
file="temp.csv", sep="\t", append=TRUE, row.names=FALSE, col.names=FALSE)
}