I need to convert some data to JSON format so I can use the data in my React.js project.
The current format of data is as follows:
["'Your Cut'"] = {
["id"] = 3400,
["tier"] = 1,
["type"] = {"Distributed"},
["limit"] = "Only granted by {{cai|Death from Below|Pyke}}.",
["maps"] = {
["sr"] = true,
["ha"] = true,
["nb"] = true,
["tt"] = true,
["cs"] = false,
},
["effects"] = {
["consume"] = "Grants {{g|text=*none*}}{{pp|100;112;140;176;220;274;300|-7+x;0+|type=target's kill bounty|color=gold}}, increased by {{g|100}} for [[First Blood]]. '''Can be used while [[dead]].'''",
},
},
["Abyssal Mask"] = {
["id"] = 8020,
["tier"] = 3,
["maps"] = {
["sr"] = true,
["ha"] = true,
["nb"] = true,
},
["menu"] = {
["tank"] = true,
},
["stats"] = {
["ah"] = 10,
["hp"] = 450,
["mr"] = 35,
},
["effects"] = {
["pass"] = {
["name"] = "Unmake",
["unique"] = true,
["description"] = "Enemy champions within {{tt|550 units|center to edge}} of you become cursed, reducing their {{as|magic resistance by 5}} {{as|(+ {{fd|1.2}}% '''bonus''' health)}}, capped at a reduction of {{as|25 magic resistance}}. Gain {{as|9 '''bonus''' magic resistance}} per cursed enemy.",
},
},
["recipe"] = {"Kindlegem", "Spectre's Cowl"},
["buy"] = 2700,
},
["Aegis of the Legion"] = {
["id"] = 3105,
["tier"] = 2,
["maps"] = {
["sr"] = true,
["ha"] = true,
["nb"] = true,
},
["menu"] = {
["tank"] = true,
["support"] = true,
},
["stats"] = {
["ah"] = 10,
["armor"] = 30,
["mr"] = 30,
},
["recipe"] = {"Null-Magic Mantle", "Cloth Armor"},
["buy"] = 1400,
},
... and so on for another 6000 lines.
I've tried doing it with replace using REGEX inside Visual Studio Code but it stopped responding and crashed the program.
What is the correct way to approach this?
Should I make a script in some language to convert the data? If so should I use REGEX or is there a smarter approach?
Thank you
You would need to know the exact syntax rules for the input. For instance, if it is possible for string literals to include special characters like newline or double quote (the delimiter), and how that would be encoded.
But for the example given, I see the following transformations that need to be done to make it valid JSON:
The square brackets around the key names should be removed
The curly braces that terminate arrays should be turned into square brackets
The whole input should be wrapped in a pair of curly braces
Trailing commas (after the last key of an object literal) should be removed
Seeing the input structure, I made several assumptions, including:
Arrays open and close on a single line;
Arrays are not nested;
Object keys are always defined on separate lines;
Key names and string values are always double quoted.
No string interpolation is performed (the double braces remain untouched)
Here is a runnable implementation of that transformation in JavaScript, which produces a result for the sample input you provided:
function toJSON(input) {
return ("{\n"
+ input.replace(/^(\s*)\[("[^"]*")](\s*)=/gm, "$1$2$3:")
.replace(/\{(.*)}(,?)[ \t]*$/gm, "[$1]$2")
+ "\n}").replace(/,(\s*\})/g, "$1");
}
// Demo on the example:
let input = `["'Your Cut'"] = {
["id"] = 3400,
["tier"] = 1,
["type"] = {"Distributed"},
["limit"] = "Only granted by {{cai|Death from Below|Pyke}}.",
["maps"] = {
["sr"] = true,
["ha"] = true,
["nb"] = true,
["tt"] = true,
["cs"] = false,
},
["effects"] = {
["consume"] = "Grants {{g|text=*none*}}{{pp|100;112;140;176;220;274;300|-7+x;0+|type=target's kill bounty|color=gold}}, increased by {{g|100}} for [[First Blood]]. '''Can be used while [[dead]].'''",
},
},
["Abyssal Mask"] = {
["id"] = 8020,
["tier"] = 3,
["maps"] = {
["sr"] = true,
["ha"] = true,
["nb"] = true,
},
["menu"] = {
["tank"] = true,
},
["stats"] = {
["ah"] = 10,
["hp"] = 450,
["mr"] = 35,
},
["effects"] = {
["pass"] = {
["name"] = "Unmake",
["unique"] = true,
["description"] = "Enemy champions within {{tt|550 units|center to edge}} of you become cursed, reducing their {{as|magic resistance by 5}} {{as|(+ {{fd|1.2}}% '''bonus''' health)}}, capped at a reduction of {{as|25 magic resistance}}. Gain {{as|9 '''bonus''' magic resistance}} per cursed enemy.",
},
},
["recipe"] = {"Kindlegem", "Spectre's Cowl"},
["buy"] = 2700,
},
["Aegis of the Legion"] = {
["id"] = 3105,
["tier"] = 2,
["maps"] = {
["sr"] = true,
["ha"] = true,
["nb"] = true,
},
["menu"] = {
["tank"] = true,
["support"] = true,
},
["stats"] = {
["ah"] = 10,
["armor"] = 30,
["mr"] = 30,
},
["recipe"] = {"Null-Magic Mantle", "Cloth Armor"},
["buy"] = 1400,
},`;
const json = toJSON(input);
console.log(JSON.parse(json));
I'm trying to add to the JsonResult object a parsed Json string, but I couldn't do it, the parser object in the browser is shown as:
"filter":[[[]],[[[],[]]]]
This is the full code
public JsonResult AjaxStandardGet(int id)
{
Standard ec = db.Standard.FirstOrDefault(s => s.IdStandard == id);
// inside filter: { "KeyDynamic1": "Value1", "KeyDynamic2": [ "AAA", "DDD"] }
var filter = JsonConvert.DeserializeObject(ec.Filter);
return Json(new
{
ec.IdStandard,
ec.Description,
filter,
ec.Observations,
Services = ec.StandardServices.Select(s => new {
s.IdStandardServices,
Tecnology = s.Tecnology?.Nombre,
ServiceType = s.ServiceType?.Description,
s.Quantity,
s.Cost,
Options = (!string.IsNullOrEmpty(s.Options) ? s.Options.Split(',') : null),
Total = s.Quantity * s.Cost
}),
Success = true
});
}
I can't create the model object because the filters are not allways the same.
I tried this:
Dictionary<string, object> filter = JsonConvert.DeserializeObject<Dictionary<string, object>>(ec.Filter);
And I get
"filter":{"KeyDynamic1":"Value1","KeyDynamic2":[[],[]]}
I suggest you to JToken or dynamic:
JToken filter = JToken.Parse(ec.Filter);
dynamic filter = JsonConvert.DeserializeObject<dynamic>(ec.Filter);
Here is working fiddle.
Update
It seems that JavaScriptSerializer is not able to do it. So you can serialize your result using Newtonsoft.Json and return it as a string:
var result = new
{
ec.IdStandard,
ec.Description,
filter,
ec.Observations,
Services = ec.StandardServices.Select(s => new {
s.IdStandardServices,
Tecnology = s.Tecnology?.Nombre,
ServiceType = s.ServiceType?.Description,
s.Quantity,
s.Cost,
Options = (!string.IsNullOrEmpty(s.Options) ? s.Options.Split(',') : null),
Total = s.Quantity * s.Cost
}),
Success = true
};
var json = JsonConvert.SerializeObject(result);
return Content(json, "application/json");
This is my response ... My dictnary name is "msg"
{
Result = {
Data = {
engine = {
service = {
data = (
{
checked = 1;
field = 1;
name = Ettersyn;
},
{
checked = 1;
field = 1;
name = "Med Alpha VP";
},
{
checked = 0;
field = 0;
name = "Med B.IMP.";
}
);
"engine_service_id" = 1;
"engine_service_name" = "4 Cyl";
};
type = {
"engine_id" = 1;
"engine_name" = Benzin;
};
};
tact = {
service = {
data = (
{
checked = 0;
field = 1;
name = "Little ettersyn";
},
{
checked = 0;
field = 1;
name = "Short ettersyn";
}
);
"tact_service_id" = 2;
"tact_service_name" = "21-50 hk";
};
type = {
"tact_id" = 1;
"tact_name" = "2 Takt";
};
};
};
status = success;
};
}
I try to to store the data like this
var arrayMain : NSMutableArray = msg.valueForKey("Result")?.valueForKey("Data") as NSMutableArray
My problem is i got "EXC_BAD_ACCESS" error While running ... Am also check allocation of array ... I try to store it in NSMutableDictionary its work perfectly ... How can i store this response in NSMutableArray ?
If you look carefully at the response data structure you'll see the problem.
This first operation: msg.valueForKey("Result") would return an dictionary with two key/values; a "Data" dictionary, and "Status" string.
Your second operation .valueForKey("Data") would return a dictionary with two key/values; an "Engine and "Tact" dictionary.
Because it's a dictionary, you can create a dictionary from it, which is what you said worked, or you're going to have to rethink how you want to store if if indeed you want it in an array, but you can't create an array directly from a dictionary - they are different data structures.
You cannot assign Dictionary object to Array, that's why it is giving the error.
But you can store that dictionary to array as:
var arrayMain : NSMutableArray = NSMutableArray()
arrayMain.addObject(msg.valueForKey("Result")?.valueForKey("Data"))
I am reading text files and creating Json objects JsValues in every iteration. I want to save them to a file at every iteration. I am using Play Framework to create JSON objects.
class Cleaner {
def getDocumentData() = {
for (i <- no_of_files) {
.... do something ...
some_json = Json.obj("text" -> LARGE_TEXT)
final_json = Json.stringify(some_json)
//save final_json here to a file
}
}
}
I tried using PrintWriter to save that json but I am getting Exception in thread "main" org.apache.spark.SparkException: Task not serializable as the error.
How should I correct this? or is there any other way I can save the JsValue?
UPDATE:
I read that the trait serializable has to be used in this case. I have the following function:
class Cleaner() extends Serializable {
def readDocumentData() {
val conf = new SparkConf()
.setAppName("linkin_spark")
.setMaster("local[2]")
.set("spark.executor.memory", "1g")
.set("spark.rdd.compress", "true")
.set("spark.storage.memoryFraction", "1")
val sc = new SparkContext(conf)
val temp = sc.wholeTextFiles("text_doc.dat)
val docStartRegex = """<DOC>""".r
val docEndRegex = """</DOC>""".r
val docTextStartRegex = """<TEXT>""".r
val docTextEndRegex = """</TEXT>""".r
val docnoRegex = """<DOCNO>(.*?)</DOCNO>""".r
val writer = new PrintWriter(new File("test.json"))
for (fileData <- temp) {
val filename = fileData._1
val content: String = fileData._2
println(s"For $filename, the data is:")
var startDoc = false // This is for the
var endDoc = false // whole file
var startText = false //
var endText = false //
var textChunk = new ListBuffer[String]()
var docID: String = ""
var es_json: JsValue = Json.obj()
for (current_line <- content.lines) {
current_line match {
case docStartRegex(_*) => {
startDoc = true
endText = false
endDoc = false
}
case docnoRegex(group) => {
docID = group.trim
}
case docTextStartRegex(_*) => {
startText = true
}
case docTextEndRegex(_*) => {
endText = true
startText = false
}
case docEndRegex(_*) => {
endDoc = true
startDoc = false
es_json = Json.obj(
"_id" -> docID,
"_source" -> Json.obj(
"text" -> textChunk.mkString(" ")
)
)
writer.write(es_json.toString())
println(es_json.toString())
textChunk.clear()
}
case _ => {
if (startDoc && !endDoc && startText) {
textChunk += current_line.trim
}
}
}
}
}
writer.close()
}
}
This is function to which I added the trait but still I am getting the same exception.
I rewrote a smaller version of it:
def foo() {
val conf = new SparkConf()
.setAppName("linkin_spark")
.setMaster("local[2]")
.set("spark.executor.memory", "1g")
.set("spark.rdd.compress", "true")
.set("spark.storage.memoryFraction", "1")
val sc = new SparkContext(conf)
var es_json: JsValue = Json.obj()
val writer = new PrintWriter(new File("test.json"))
for (i <- 1 to 10) {
es_json = Json.obj(
"_id" -> i,
"_source" -> Json.obj(
"text" -> "Eureka!"
)
)
println(es_json)
writer.write(es_json.toString() + "\n")
}
writer.close()
}
This function works fine with and also without serializable. I cannot understand what's happening?
EDIT: First answer made on phone.
It's not your main class that needs to be serializable but the class you use in the rdd processing loop in this case inside for (fileData <- temp)
It needs to be serializable because the spark data is on multiple partitions that may be on multiple computers. So the functions you apply to this data need to be serializable so you can send them to the other computer where they will be executed in parallel.
PrintWriter cannot be serializable since it refers to a file that is only available from the original computer. Hence the serializaion error.
To write your data on the computer initializing the spark process. You need to take the data that is all over the cluster and bring it to your machine then write it.
To do that you can either collect the result. rdd.collect() and that will take all the data from the cluster and put it in your driver thread memory. Then you can write it to a file using the PrintWriter.
like this:
temp.flatMap { fileData =>
val filename = fileData._1
val content: String = fileData._2
println(s"For $filename, the data is:")
var startDoc = false // This is for the
var endDoc = false // whole file
var startText = false //
var endText = false //
var textChunk = new ListBuffer[String]()
var docID: String = ""
var es_json: JsValue = Json.obj()
var results = ArrayBuffer[String]()
for (current_line <- content.lines) {
current_line match {
case docStartRegex(_*) => {
startDoc = true
endText = false
endDoc = false
}
case docnoRegex(group) => {
docID = group.trim
}
case docTextStartRegex(_*) => {
startText = true
}
case docTextEndRegex(_*) => {
endText = true
startText = false
}
case docEndRegex(_*) => {
endDoc = true
startDoc = false
es_json = Json.obj(
"_id" -> docID,
"_source" -> Json.obj(
"text" -> textChunk.mkString(" ")
)
)
results.append(es_json.toString())
println(es_json.toString())
textChunk.clear()
}
case _ => {
if (startDoc && !endDoc && startText) {
textChunk += current_line.trim
}
}
}
}
results
}
.collect()
.foreach(es_json => writer.write(es_json))
If the result is too large for the driver thread memory you can use the saveAsTextFile function that will stream each partition to your drive. In this second case the path you give as argument will be made into a folder and each partition of your rdd will be written to a numbered file in it.
like this:
temp.flatMap { fileData =>
val filename = fileData._1
val content: String = fileData._2
println(s"For $filename, the data is:")
var startDoc = false // This is for the
var endDoc = false // whole file
var startText = false //
var endText = false //
var textChunk = new ListBuffer[String]()
var docID: String = ""
var es_json: JsValue = Json.obj()
var results = ArrayBuffer[String]()
for (current_line <- content.lines) {
current_line match {
case docStartRegex(_*) => {
startDoc = true
endText = false
endDoc = false
}
case docnoRegex(group) => {
docID = group.trim
}
case docTextStartRegex(_*) => {
startText = true
}
case docTextEndRegex(_*) => {
endText = true
startText = false
}
case docEndRegex(_*) => {
endDoc = true
startDoc = false
es_json = Json.obj(
"_id" -> docID,
"_source" -> Json.obj(
"text" -> textChunk.mkString(" ")
)
)
results.append(es_json.toString())
println(es_json.toString())
textChunk.clear()
}
case _ => {
if (startDoc && !endDoc && startText) {
textChunk += current_line.trim
}
}
}
}
results
}
.saveAsTextFile("test.json")
I am getting the following data in JSON format by the Facebook graph API. However it looks like it is in invalid JSON format. The JSON formatter says that the error is in line 1. What is this error ?
{
data = (
{
application = {
id = 2409997254;
name = Likes;
};
"created_time" = "2013-05-05T07:51:41+0000";
from = {
id = 100000347121257;
name = "Rishab Gulati";
};
id = "notif_746853089_182660043";
link = "http://www.facebook.com/photo.php?fbid=10151457234013090&set=a.427064503089.223857.746853089&type=1";
object = "<null>";
title = "Rishab Gulati, Deepika Gurnani and 2 other people like your photo.";
to = {
id = 746853089;
name = "Ashish Agarwal";
};
unread = 1;
"updated_time" = "2013-05-05T10:48:33+0000";
}
);
paging = {
next = "https://graph.facebook.com/746853089/notifications?format=json&access_token=**xxxxxxxxxx";
previous = "https://graph.facebook.com/746853089/notifications?format=json&access_token=**SNIP**&limit=5000&since=1367740301&__paging_token=notif_746853089_182660043&__previous=1";
};
summary = {
"unseen_count" = 1;
"updated_time" = "2013-05-05T10:48:33+0000";
};
}
This data is incorrect and isn't what Facebook supplies
{
"data": [
{
Where as your JSON above shows
{
data = (
{
There is a ( when there should be a {
Also consider not giving your access token out in public.