How convert tsv to Json - json

I want to make a dynamic graph based on a json file. I have seen many examples with tsv but I donot how to convert it to json.
That is the part that I want to change from tsv to json but I donot know how!
d3.tsv("data/data.tsv", function(error, data) {
data.forEach(function(d) {
d.date = parseDate(d.date);
d.close = +d.close;
});
when I use
d3.json("data/data.json", function(data) {
data.forEach(function d) {
d.date = parseDate(d.date);
d.close = +d.close;
}
});
it gives this error: Uncaught type error: cannot call method 'forEach' of undefined!
Thanks for your suggestions :)

try to do something like this
d3.json("data/data.json", function(data) {
data.forEach(function d) {
d.date = parseDate(d.date);
d.close = +d.close;
}
});
d3.js have support for json, https://github.com/mbostock/d3/wiki/Requests

The syntax around your forEach is a little off; try this instead:
d3.json("data/data.json", function(data) {
data.forEach(function(d) {
d.date = parseDate(d.date);
d.close = +d.close;
});
});
(As Felix points out, this will only work if your JSON object is defined and is an array)

Here a small code where you'll be able to convert tsv to json. It could help you...
ps : here is typescript, but you can easily convert it to vanilla javascript ;)
// Set bunch of datas into format object
tsvToJson(datas: string): Array<Object>{
// Separate each lines
let array_datas = datas.split(/\r\n|\r|\n/g);
// Separate each values into each lines
var detailed_datas = [];
for(var i = 0; i < array_datas.length; i++){
detailed_datas.push(array_datas[i].split("\t"));
}
// Create index
var index = [];
var last_index = ""; // If the index we're reading is equal to "", it mean it might be an array so we take the last index
for(var i = 0; i < detailed_datas[0].length; i++){
if(detailed_datas[0][i] == "") index.push(last_index);
else {
index.push(detailed_datas[0][i]);
last_index = detailed_datas[0][i];
}
}
// Separate data from index
detailed_datas.splice(0, 1);
// Format data
var formated_datas = [];
for(var i = 0; i < detailed_datas.length; i++){
var row = {};
for(var j = 0; j < detailed_datas[i].length; j++){
// Check if value is empty
if(detailed_datas[i][j] != ""){
if(typeof row[index[j]] == "object"){
// it's already set as an array
row[index[j]].push(detailed_datas[i][j]);
} else if(row[index[j]] != undefined){
// Already have a value, so it might be an array
row[index[j]] = [row[index[j]], detailed_datas[i][j]];
} else {
// It's empty for now, so let's say first that it's a string
row[index[j]] = detailed_datas[i][j];
}
}
}
formated_datas.push(row);
}
console.log(formated_datas); // #TODO : remove this
return formated_datas;
}

I transpile and resume Wetteren's code:
convertTSVtoJSON(tsvData) {
const formattedData = tsvData.split(/\r\n|\r|\n/g).filter(e => !!e).map((parsedEntry) => parsedEntry.split("\t"));
const tsvHeaders = formattedData.shift();
return formattedData.map(formattedEntry => {
{
return tsvHeaders.reduce((jsonObject, heading, position) => {
jsonObject[heading] = formattedEntry[position];
return jsonObject;
}, {});
}
});
}

Related

Angular 2 - Parsing Excel worksheet to Json

I have an Excel file with the following content:
Inside my component.ts, I extract the Excel's content as follow:
var testUrl= "excel.xlsx";
var oReq = new XMLHttpRequest();
oReq.open("GET", testUrl, true);
oReq.responseType = "arraybuffer";
oReq.onload = function(e) {
var arraybuffer = oReq.response;
var data = new Uint8Array(arraybuffer);
var arr = new Array();
for(var i = 0; i != data.length; ++i){
arr[i] = String.fromCharCode(data[i]);
}
var bstr = arr.join("");
var workbook = XLSX.read(bstr, {type:"binary"});
var first_sheet_name = workbook.SheetNames[0];
var worksheet = workbook.Sheets[first_sheet_name];
var json = XLSX.utils.sheet_to_json(workbook.Sheets[workbook.SheetNames[0]], {header:1, raw:true});
var jsonOut = JSON.stringify(json);
console.log("test"+jsonOut);
}
oReq.onerror = function(e) {
console.log(e);
}
oReq.send();
XLSX.utils.sheet_to_json will format JSON as follow:
However, I would like the JSON to be as follow:
Most probably I would need to manually create the JSON, but can anyone help me point to the direction on how I can accomplish this?
In your case we need to modify the JSON data by looping over XLSX.utils.sheet_to_json JSON object:
// This object will contain the data in the format we want
var finalObj = { "object": []};
// Variables to track where to insert the data
var locIndex, firstCondIndex, secondCondIndex,
lockey, firstCondKey, secondCondkey;
// We need to initialize all indexes to -1 so that on first time we can get 0, as arrays start with 0 in javascript
locIndex = -1;
// here obj is XLSX.utils.sheet_to_json
obj.object.map((value, index) => {
// we don't want to consider name of columns which is first element of array
if(!index) return;
// Go inside only if not null
if(value[0]) {
// For Location
finalObj.object.push(createObj(value[0]));
locIndex++;
// We also need to store key names to push it's children
lockey = value[0];
firstCondIndex = -1;
}
if(value[1]) {
// For First Condition
finalObj.object[locIndex][lockey].push(createObj(value[1]));
firstCondIndex++;
firstCondKey = value[1];
secondCondIndex = -1;
}
if(value[2]) {
// For Second Condition
finalObj.object[locIndex][lockey][firstCondIndex][firstCondKey].push(createObj(value[2]));
secondCondIndex++;
secondCondkey = value[2];
}
if(value[3]) {
// For Products
// We just push the string
finalObj.object[locIndex][lockey][firstCondIndex][firstCondKey][secondCondIndex][secondCondkey].push(value[3]);
}
});
function createObj(val) {
// We need to initialize blank array so we can push the children of that element later on
var obj = {};
obj[val] = [];
return obj;
}
console.log(finalObj);

Convert a javascript array to specific json format

I have this JSON.stringify(array) result: ["id:1x,price:150","id:2x,price:200"] and I have to convert it in a json format similar to this (it would be sent to php):
[{
"id":"1x",
"price":150
},
{
"id":"2x",
"price":200
}]
Please help.
You can convert the structure of your data before using JSON.stringify, for example you can convert the data to a list of objects like so:
var strItems = ["id:1x,price:150", "id:2x,price:200"];
var objItems = [];
for (var i = 0; i < strItems.length; i++) {
var pairs = strItems[i].split(",");
objItems.push({
id: pairs[0].split(':')[1],
price: parseInt(pairs[1].split(':')[1])
});
}
Before calling JSON.stringify to get the result you're after:
var json = JSON.stringify(objItems);
console.log(json);
JSON.stringify(array) only does on layer, but you can change the stringify function to work for multiple layers with this function written by JVE999 on this post:
(function(){
// Convert array to object
var convArrToObj = function(array){
var thisEleObj = new Object();
if(typeof array == "object"){
for(var i in array){
var thisEle = convArrToObj(array[i]);
thisEleObj[i] = thisEle;
}
}else {
thisEleObj = array;
}
return thisEleObj;
};
var oldJSONStringify = JSON.stringify;
JSON.stringify = function(input){
if(oldJSONStringify(input) == '[]')
return oldJSONStringify(convArrToObj(input));
else
return oldJSONStringify(input);
};
})();
Once you have declared this, then you can call JSON.stringify(array)

How to check if properties in a JSON file are ordered?

I have a big json file which is changed quite often (translation values). I need to check whether properties in this json are sorted alphabetically. Perfectly I'd like to use gulp for that and run this check in CI environment so when properties aren't ordered an error is reported.
I've searched for it but haven't found anything. Are you aware of any existing solution dealing with this problem?
You could do it purely with node.js and hook it into your gulp build process easily enough:
var fs = require('fs');
fs.readFile("data.json", { encoding: 'utf8' }, function(err, data) {
if (err) throw err;
var dataObj = JSON.parse(data).mystuff;
//check if it's ordered
for (var i = 1; i < dataObj.length; i++) {
if (dataObj[i].toLowerCase() < dataObj[i-1].toLowerCase()) { //note that depending on what it is you're ordering, this might need to change...
throw new Error('THIS IS OUT OF ORDER! I TOLD YOU! I TOLD YOU!!!! KEEP IT ORDERED! AND WHAT DID YOU DO?!!! UNACCEPTABLE!!!');
}
}
console.log('Ordered correctly! Good work!');
});
And the data.json that I used to test it, ordered incorrectly:
{
"mystuff": [
"Super",
"Duper",
"Happy",
"Fun"
]
}
And ordered correctly:
{
"mystuff": [
"Duper",
"Fun",
"Happy",
"Super"
]
}
And if you want to take it a step further and have the process automatically re-order the file for you, you could do something like this:
var fs = require('fs');
var fileName = 'data.json';
var fileOptions = { encoding: 'utf8' };
fs.readFile(fileName, fileOptions, function(err, data) {
if (err) throw err;
debugger;
var dataObj = JSON.parse(data);
var dataObjOrdered = orderObj(dataObj);
var jsonOrdered = JSON.stringify(dataObjOrdered, null, 4);
fs.writeFile(fileName, jsonOrdered, fileOptions, function(err) {
if (err) throw err;
console.log('JSON data fields are ordered huzzah!');
});
});
/**
* Returns an object with the same data, but with its fields ordered.
* Note that we leave array values in the same order they're already in
*/
function orderObj(obj) {
if (!isObject(obj)) {
return obj;
}
//arrays are left alone. if you want to order them too, do it inside this block
if (Array.isArray(obj)) {
return obj;
}
//load up an array containing all our object's fields
var fields = [];
for (var field in obj) {
fields.push(field);
}
//sort the fields alphabetically (remember to do this compare case-insensitively since utf-8 capital letters come before lowercase letters)
fields.sort(function (a, b) {
a = a.toLowerCase();
b = b.toLowerCase();
if (a > b) {
return 1;
}
if (a < b) {
return -1;
}
return 0;
});
//return the new object with ordered fields, recursively ordering sub objects' fields
var result = {};
for(var i = 0; i < fields.length; i++) {
var field = fields[i];
var value = obj[field];
result[field] = orderObj(value);
}
return result;
}
function isObject(value) {
// http://jsperf.com/isobject4
return value !== null && typeof value === 'object';
}

How to pass a whole dojox.grid.DataGrid store(items json data) to servlet?

I have a button on page - when clicked, it passes all the data to the servlet that could update each row data. My question is how to pass the whole store to the servlet as json data? Is there any easy way? Thanks
Here is some code I wrote to get the store to an object. Then it can be converted to JSON using dojo.toJson(obj);. I learned about this from the dojotoolkit website originally. (Give credit where credit is due). I realize this code is huge and nasty. When I looked for a better way about a year back I could not find one.
JsonHelper.storeToObject = function(store) {
var object = [];
var index = -1;
store.fetch({
onItem : function(item, request) {
object[++index] = JsonHelper.itemToObject(store, item);
}
});
return object;
};
JsonHelper.itemToObject = function(store, item) {
// store:
// The datastore the item came from.
// item:
// The item in question.
var obj = {};
if (item && store) {
// Determine the attributes we need to process.
var attributes = store.getAttributes(item);
if (attributes && attributes.length > 0) {
var i;
for (i = 0; i < attributes.length; i++) {
var values = store.getValues(item, attributes[i]);
if (values) {
// Handle multivalued and single-valued attributes.
if (values.length > 1) {
var j;
obj[attributes[i]] = [];
for (j = 0; j < values.length; j++) {
var value = values[j];
// Check that the value isn't another item. If
// it is, process it as an item.
if (store.isItem(value)) {
obj[attributes[i]].push(itemToObject(store,
value));
} else {
obj[attributes[i]].push(value);
}
}
} else {
if (store.isItem(values[0])) {
obj[attributes[i]] = itemToObject(store,
values[0]);
} else {
obj[attributes[i]] = values[0];
}
}
}
}
}
}
return obj;
};

Can I increase QUOTA_BYTES_PER_ITEM in Chrome?

Is there any way to increase the chrome.storage.sync.QUOTA_BYTES_PER_ITEM ?
For me, the default 4096 Bytes is a little bit short.
I tried to execute
chrome.storage.sync.QUOTA_BYTES_PER_ITEM = 8192;
However, it seems that the actual limit doesn't change.
How can I do this?
No, QUOTA_BYTES_PER_ITEM is there for reference only; it is not a settable value. You could use the value of QUOTA_BYTES_PER_ITEM to split an item up into multiple items, though:
function syncStore(key, objectToStore, callback) {
var jsonstr = JSON.stringify(objectToStore);
var i = 0;
var storageObj = {};
// split jsonstr into chunks and store them in an object indexed by `key_i`
while(jsonstr.length > 0) {
var index = key + "_" + i++;
// since the key uses up some per-item quota, see how much is left for the value
// also trim off 2 for quotes added by storage-time `stringify`
var valueLength = chrome.storage.sync.QUOTA_BYTES_PER_ITEM - index.length - 2;
// trim down segment so it will be small enough even when run through `JSON.stringify` again at storage time
var segment = jsonstr.substr(0, valueLength);
while(JSON.stringify(segment).length > valueLength)
segment = jsonstr.substr(0, --valueLength);
storageObj[index] = segment;
jsonstr = jsonstr.substr(valueLength);
}
// store all the chunks
chrome.storage.sync.set(storageObj, callback);
}
Then write an analogous fetch function that fetches by key and glues the object back together.
just modify answer of #apsilliers
function syncStore(key, objectToStore) {
var jsonstr = JSON.stringify(objectToStore);
var i = 0;
var storageObj = {};
// split jsonstr into chunks and store them in an object indexed by `key_i`
while(jsonstr.length > 0) {
var index = key + "_" + i++;
// since the key uses up some per-item quota, see how much is left for the value
// also trim off 2 for quotes added by storage-time `stringify`
const maxLength = chrome.storage.sync.QUOTA_BYTES_PER_ITEM - index.length - 2;
var valueLength = jsonstr.length;
if(valueLength > maxLength){
valueLength = maxLength;
}
// trim down segment so it will be small enough even when run through `JSON.stringify` again at storage time
//max try is QUOTA_BYTES_PER_ITEM to avoid infinite loop
var segment = jsonstr.substr(0, valueLength);
for(let i = 0; i < chrome.storage.sync.QUOTA_BYTES_PER_ITEM; i++){
const jsonLength = JSON.stringify(segment).length;
if(jsonLength > maxLength){
segment = jsonstr.substr(0, --valueLength);
}else {
break;
}
}
storageObj[index] = segment;
jsonstr = jsonstr.substr(valueLength);
}
also function to read each partition and merge again
function syncGet(key, callback) {
chrome.storage.sync.get(key, (data) => {
console.log(data[key]);
console.log(typeof data[key]);
if(data != undefined && data != "undefined" && data != {} && data[key] != undefined && data[key] != "undefined"){
const keyArr = new Array();
for(let i = 0; i <= data[key].count; i++) {
keyArr.push(`${data[key].prefix}${i}`)
}
chrome.storage.sync.get(keyArr, (items) => {
console.log(data)
const keys = Object.keys( items );
const length = keys.length;
let results = "";
if(length > 0){
const sepPos = keys[0].lastIndexOf("_");
const prefix = keys[0].substring(0, sepPos);
for(let x = 0; x < length; x ++){
results += items[`${prefix }_${x}`];
}
callback(JSON.parse(results));
return;
}
callback(undefined);
});
} else {
callback(undefined);
}
});
}
it tested and it works for my case
this is a better version of #uncle bob's functions, working with manifest v3 (you can use it just like how you can use the normal sync.set or sync.get function)
NOTE: it only works with JSONs (arrays and objects) since a string shouldn't be that long
let browserServices;
if (typeof browser === "undefined") {
browserServices = chrome;
} else {
browserServices = browser;
}
function syncSet(obj = {}) {
return new Promise((resolve, reject) => {
var storageObj = {};
for (let u = 0; u < Object.keys(obj).length; u++) {
const key = Object.keys(obj)[u];
const objectToStore = obj[key]
var jsonstr = JSON.stringify(objectToStore);
var i = 0;
// split jsonstr into chunks and store them in an object indexed by `key_i`
while (jsonstr.length > 0) {
var index = key + "USEDTOSEPERATE" + i++;
// since the key uses up some per-item quota, see how much is left for the value
// also trim off 2 for quotes added by storage-time `stringify`
const maxLength = browserServices.storage.sync.QUOTA_BYTES_PER_ITEM - index.length - 2;
var valueLength = jsonstr.length;
if (valueLength > maxLength) {
valueLength = maxLength;
}
// trim down segment so it will be small enough even when run through `JSON.stringify` again at storage time
//max try is QUOTA_BYTES_PER_ITEM to avoid infinite loop
var segment = jsonstr.substring(0, valueLength);
var jsonLength = JSON.stringify(segment).length;
segment = jsonstr.substring(0, valueLength = (valueLength - (jsonLength - maxLength) - 1));
for (let i = 0; i < browserServices.storage.sync.QUOTA_BYTES_PER_ITEM; i++) {
jsonLength = JSON.stringify(segment).length;
if (jsonLength > maxLength) {
segment = jsonstr.substring(0, --valueLength);
} else {
break;
}
}
storageObj[index] = segment;
jsonstr = jsonstr.substring(valueLength, Infinity);
}
}
chrome.storage.sync.set(storageObj).then(() => {
resolve()
})
})
}
function syncGet(uniqueKeys = []) {
return new Promise((resolve, reject) => {
browserServices.storage.sync.get(null).then((data) => {
const keyArr = Object.keys(data).filter(e => uniqueKeys.filter(j => e.indexOf(j) == 0).length > 0)
browserServices.storage.sync.get(keyArr).then((items) => {
var results = {};
for (let i = 0; i < uniqueKeys.length; i++) {
const uniqueKey = uniqueKeys[i];
const keysFiltered = keyArr.filter(e => e.split("USEDTOSEPERATE")[0] == uniqueKey)
if (keysFiltered.length > 0) {
results[uniqueKey] = ""
for (let x = 0; x < keysFiltered.length; x++) {
results[uniqueKey] += items[`${keysFiltered[x]}`];
}
results[uniqueKey] = JSON.parse(results[uniqueKey])
}
}
resolve(results)
});
});
})
}
example of usage:
syncSet({
"keyTest": ["a lot of text"],
"keyTest1": ["a lot of text"]
}
)
syncGet(["keyTest","keyTest1"]).then(results=>console.log(results))
// {keyTest:["a lot of text"],keyTest1:["a lot of text"]}