Working with nested lists in R - json

I have dataset in json format. When I pull this data to a variable in R using jsonlite::fromJson, a list is created. This list further contains another list, which in turn contains another list and final list has other objects.
How do I make a dataframe out of this data?
Sample dataset:
{
"AG2": {
"-KB89nmaAHtjvIHl6X6I": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 0,
"score": 20,
"timeBonus": 95,
"timestamp": "Mon Feb 22 2016 18:59:17 GMT+0530 (India Standard Time)",
"totalScore": 20
},
"-KB89q8EJ-rYvjzVcIiQ": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 1,
"score": 39,
"timeBonus": 93,
"timestamp": "Mon Feb 22 2016 18:59:27 GMT+0530 (India Standard Time)",
"totalScore": 59
},
"-KB89s7SVjq0YZQ5XrOt": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 2,
"score": 59,
"timeBonus": 95,
"timestamp": "Mon Feb 22 2016 18:59:35 GMT+0530 (India Standard Time)",
"totalScore": 118
},
"-KB89xsifcqbxCDE9Ygt": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 3,
"score": 78,
"timeBonus": 93,
"timestamp": "Mon Feb 22 2016 18:59:59 GMT+0530 (India Standard Time)",
"totalScore": 196
},
"-KB8A-85L2P9nGmii3L_": {
"attempts": 1,
"isCorrect": "The output value should be your name",
"questionIndex": 4,
"score": 0,
"timeBonus": 95,
"timestamp": "Mon Feb 22 2016 19:00:08 GMT+0530 (India Standard Time)",
"totalScore": 196
},
"-KB8A1JTmwee2a5gRkLq": {
"attempts": 1,
"isCorrect": true,
"questionIndex": 4,
"score": 85,
"timeBonus": 87,
"timestamp": "Mon Feb 22 2016 19:00:17 GMT+0530 (India Standard Time)",
"totalScore": 281
}
},
"AKD": {
"-KB88bypMqEu3Y2zOEKD": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 0,
"score": 19,
"timeBonus": 90,
"timestamp": "Mon Feb 22 2016 18:54:07 GMT+0530 (India Standard Time)",
"totalScore": 19
},
"-KB88grpZUs3j3J4SNWc": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 1,
"score": 37,
"timeBonus": 85,
"timestamp": "Mon Feb 22 2016 18:54:27 GMT+0530 (India Standard Time)",
"totalScore": 56
},
"-KB88jYngOMf7chyXUch": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 2,
"score": 58,
"timeBonus": 91,
"timestamp": "Mon Feb 22 2016 18:54:38 GMT+0530 (India Standard Time)",
"totalScore": 114
},
"-KB88mtpsbwp6GD4VQ0f": {
"attempts": 0,
"isCorrect": true,
"questionIndex": 3,
"score": 76,
"timeBonus": 89,
"timestamp": "Mon Feb 22 2016 18:54:52 GMT+0530 (India Standard Time)",
"totalScore": 190
},
"-KB88z1nL0Ckhn5A5Lsq": {
"attempts": 1,
"isCorrect": "The output value should be your name",
"questionIndex": 4,
"score": 0,
"timeBonus": 60,
"timestamp": "Mon Feb 22 2016 18:55:41 GMT+0530 (India Standard Time)",
"totalScore": 190
}
}
}
Required dataframe:
User | TimeStamp | Question Index | Attempts | TimeBonus | isCorrect | Score | Total
Thanks.

Related

Finding the error in the script that runs a FIFO/LIFO calculation on a transactional history of an investment tracker

Introduction
I have a spreadsheet and the idea of this spreadsheet is that any user can input his or her transaction history in the sheet "history" and based on this history, an overview is provided of a person's realized gains (=they have sold it in the past) and their unrealized gains (= never sold it). A script is run which then creates a report (sheet: Report) with the necessary data. A query table summarises this data in the summary of RG report sheet. Based on this summary, the position sheet is created, which shows all the current positions held by the user. The user can chose to run the script by going to the menu CoinAtlas>Build Report>LIFO or HIFO. Please see here the sample sheet.
What is FIFO and LIFO?
These are accepted accounting methods to calculate realized gains. FIFO (=first in, first out) & LIFO (=last in, first out). An example of FIFO: You buy 2 stocks of company A for 10 dollars each at date Y. At date Y+1 you buy 2 more stocks of company A for 15 dollars each. You hold 4 stocks of company A. You decide to sell 3 stocks of company A for 20 dollars each. Profit is then calculated as (3 * 20-(2 * 10+1 * 15)) = 60 - 35 = 25 dollars. The same scenario LIFO would be: 60 - 40 = 20 dollars.
Description of the problem
When building a report, for whatever reason, the stock MSTR and the cryptocurrency THETA is ignored and not shown in the output sheet "Report". The script was written by an experienced programmer who unfortunately no longer responds to this issue. With my beginner's skills in javascript I cannot seem to find out why.
The FIFO script calculation
function createReport_LIFO () {
const ss = SpreadsheetApp.getActiveSpreadsheet();
const inputSheet = ss.getSheetByName('History');
const report = ss.getSheetByName('Report');
const data = inputSheet.getRange(2, 1, inputSheet.getLastRow() - 1, 9).getValues();
const filterd = data.filter(row => row[2] == 'Buy' || row[2] == 'Sell');
const sorted = filterd.sort((a, b) => {
if (a[0] < b[0]) return 1
if (a[0] > b[0]) return -1
else return 0
})
//console.log(sorted);
const securityObject = sorted.reduce((acc, curr) => {
if (curr[1] in acc) {
if (curr[2] in acc[curr[1]]) {
acc[curr[1]][curr[2]].push(curr)
} else {
acc[curr[1]] = { ...acc[curr[1]], [curr[2]]: [curr] }
}
} else {
acc[curr[1]] = { [curr[2]]: [curr] }
}
return acc
}, {});
console.log(JSON.stringify(securityObject));
//console.log(securityObject);
const objects = [];
Object.keys(securityObject).forEach(ticker => {
const tic = securityObject[ticker];
let index = 0;
try {
tic.Sell.forEach(sell => {
const [date, security, , quanity, total, , account, ] = sell;
let totalBuy = 0;
let remainder = quanity;
do {
let [, , , buyQuanity, , buyPrice] = tic.Buy[index];
if (buyQuanity < remainder) {
totalBuy += (buyQuanity * buyPrice);
remainder -= buyQuanity;
index++;
} else {
totalBuy += (remainder * buyPrice);
securityObject[ticker].Buy[index][3] = (buyQuanity - remainder);
buyQuanity -= remainder;
remainder = 0;
if (buyQuanity <= 0) {
index++;
}
}
} while (remainder > 0);
objects.push({
date,
security,
account,
quanity,
totalBuy,
total,
result: total - totalBuy
})
})
} catch (err) {
console.log(err)
}
})
const convertToSheetsArray = [["Sell Date", "Security", "Account", "Quantity Sold", "Total Buy", "Total Sell", "Result"]]
objects.forEach(obj => convertToSheetsArray.push(Object.values(obj)));
report.getDataRange().clearContent();
report.getRange(1, 1, convertToSheetsArray.length, 7).setValues(convertToSheetsArray);
}
Output of securityObject
Logging output too large. Truncating output. {"ETH":{"Buy":[["2021-10-26T22:00:00.000Z","ETH","Buy",0.087541,301.25,3441.2446739242187,"Bitvavo","Cryptocurrency",""],["2021-07-11T22:00:00.000Z","ETH","Buy",1.19,2113.84,1776.3361344537818,"Bitvavo","Cryptocurrency",""],["2021-07-09T22:00:00.000Z","ETH","Buy",0.109074,195.55,1792.8195536974897,"Binance","Cryptocurrency",""],["2021-07-09T22:00:00.000Z","ETH","Buy",0.0529245,94.95,1794.0651305161127,"Binance","Cryptocurrency",""],["2021-07-09T22:00:00.000Z","ETH","Buy",0.0443436,81.22,1831.6059138184542,"Binance","Cryptocurrency",""],["2021-07-09T22:00:00.000Z","ETH","Buy",0.31543836,576.66,1828.1226164122843,"Binance","Cryptocurrency",""],["2021-07-05T22:00:00.000Z","ETH","Buy",0.19039196,498.75,2619.595911508028,"Bitvavo","Cryptocurrency",""],["2021-07-05T22:00:00.000Z","ETH","Buy",0.57131,1116.81,1954.823125798603,"Binance","Cryptocurrency",""],["2021-05-18T22:00:00.000Z","ETH","Buy",0.1036,250,2413.127413127413,"Binance","Cryptocurrency",""]]},"MATIC":{"Sell":[["2021-10-21T22:00:00.000Z","MATIC","Sell",37.14,61.82,1.6645126548196014,"Binance","Cryptocurrency",""]],"Buy":[["2021-05-18T22:00:00.000Z","MATIC","Buy",37.14,65,1.7501346257404415,"Binance","Cryptocurrency",""]]},"XTZ":{"Buy":[["2021-10-21T22:00:00.000Z","XTZ","Buy",8.6,58.82,6.839534883720931,"Binance","Cryptocurrency",""],["2021-09-25T22:00:00.000Z","XTZ","Buy",35.17,200,5.686664771111743,"Bitvavo","Cryptocurrency",""],["2021-09-11T22:00:00.000Z","XTZ","Buy",35.777,200,5.590183637532493,"Bitvavo","Cryptocurrency",""]]},"TFUEL":{"Sell":[["2021-10-20T22:00:00.000Z","TFUEL","Sell",754,182,0.2413793103448276,"Binance","Cryptocurrency",""]],"Buy":[["2021-07-04T22:00:00.000Z","TFUEL","Buy",754,261,0.34615384615384615,"Binance","Cryptocurrency",""]]},"THETA":{"Sell":[["2021-10-20T22:00:00.000Z","THETA","Sell",81.4,397.73,4.886117936117936,"Binance","Cryptocurrency",""]],"Buy":[["2021-03-26T23:00:00.000Z","THETA","Buy",19.5,200.56,10.285128205128204,"Binance","Cryptocurrency",""],["2021-03-20T23:00:00.000Z","THETA","Buy",26,200,7.6923076923076925,"Binance","Cryptocurrency",""],["2021-03-19T23:00:00.000Z","THETA","Buy",35.9,249.81,6.95849582172702,"Binance","Cryptocurrency",""]]},"ONE":{"Buy":[["2021-10-20T22:00:00.000Z","ONE","Buy",2768,579.73,0.2094400289017341,"Binance","Cryptocurrency",""],["2021-09-11T22:00:00.000Z","ONE","Buy",1241,200,0.16116035455278002,"Binance","Cryptocurrency",""]]},"ALGO":{"Buy":[["2021-09-25T22:00:00.000Z","ALGO","Buy",145.18,200,1.3776002204160351,"Bitvavo","Cryptocurrency",""]]},"UPST":{"Buy":[["2021-08-09T22:00:00.000Z","UPST","Buy",2.8786672,336,116.7206824046906,"Trading212","Stock",""]]},"AXS":{"Buy":[["2021-08-08T22:00:00.000Z","AXS","Buy",3,106,35.333333333333336,"Binance","Cryptocurrency",""]]},"safemars":{"Sell":[["2021-08-08T22:00:00.000Z","safemars","Sell",206130076,16,7.762089021885385e-8,"Binance","Cryptocurrency",""]],"Buy":[["2021-04-19T22:00:00.000Z","safemars","Buy",206130076,200,9.70261127735673e-7,"Binance","Cryptocurrency",""]]},"safemoon":{"Sell":[["2021-08-08T22:00:00.000Z","safemoon","Sell",55800000,90,0.0000016129032258064516,"Binance","Cryptocurrency",""]],"Buy":[["2021-04-19T22:00:00.000Z","safemoon","Buy",55800000,300,0.000005376344086021505,"Binance","Cryptocurrency",""]]},"FRA:C36":{"Sell":[["2021-07-13T22:00:00.000Z","FRA:C36","Sell",358,343.68,0.96,"Bitvavo","Stock",""]],"Buy":[["2021-03-18T23:00:00.000Z","FRA:C36","Buy",358,253.9,0.7092178770949721,"DeGiro","Stock",""]]},"BTC":{"Sell":[["2021-07-11T22:00:00.000Z","BTC","Sell",0.038237,1116.81,29207.57381593744,"Binance","Cryptocurrency",""],["2021-07-04T22:00:00.000Z","BTC","Sell",0.00537222,257,47838.69610700977,"Binance","Cryptocurrency",""],["2021-07-04T22:00:00.000Z","BTC","Sell",0.00543778,261,47997.528403135104,"Binance","Cryptocurrency",""],["2021-07-04T22:00:00.000Z","BTC","Sell",0.00533832,261,48891.78617992178,"Binance","Cryptocurrency",""],["2021-07-04T22:00:00.000Z","BTC","Sell",0.00550857,261,47380.71768172139,"Binance","Cryptocurrency",""],["2021-07-04T22:00:00.000Z","BTC","Sell",0.00008389000000000313,0,0,"Bitvavo","Cryptocurrency",""],["2021-06-06T22:00:00.000Z","BTC","Sell",0.038276,1116.81,29177.81377364406,"Binance","Cryptocurrency",""],["2021-05-02T22:00:00.000Z","BTC","Sell",0.01032703,498.75,48295.58934175654,"Bitvavo","Cryptocurrency",""],["2021-04-13T22:00:00.000Z","BTC","Sell",0.18413661999999997,9757,52987.83044893515,"Bitvavo","Cryptocurrency",""]],"Buy":[["2021-06-23T22:00:00.000Z","BTC","Buy",0.038276,1000,26126.03197826314,"Binance","Cryptocurrency",""],["2021-04-14T22:00:00.000Z","BTC","Buy",0.18428,9757,52946.60299544172,"Bitvavo","Cryptocurrency",""],["2021-03-22T23:00:00.000Z","BTC","Buy",0.00094112,43.52,46242.77456647399,"Bitvavo","Cryptocurrency",""],["2021-02-21T23:00:00.000Z","BTC","Buy",0.025382,1000,39397.99858167205,"Bitvavo","Cryptocurrency",""],["2021-02-21T23:00:00.000Z","BTC","Buy",0.046151,1770.3,38358.86546337024,"Bitvavo","Cryptocurrency",""],["2021-02-21T23:00:00.000Z","BTC","Buy",0.046339,2003,43224.929325190446,"Bitvavo","Cryptocurrency",""],["2021-02-16T23:00:00.000Z","BTC","Buy",0.048543,2045.36,42135.0143172033,"Bitvavo","Cryptocurrency",""],["2021-02-07T23:00:00.000Z","BTC","Buy",0.0027187,100,36782.28565123036,"Bitvavo","Cryptocurrency",""],["2021-01-23T23:00:00.000Z","BTC","Buy",0.0077802,200,25706.2800442148,"Bitvavo","Cryptocurrency",""],["2021-01-10T23:00:00.000Z","BTC","Buy",0.0019139,50,26124.66691049689,"Bitvavo","Cryptocurrency",""],["2021-01-09T23:00:00.000Z","BTC","Buy",0.0043677,134,30679.7628042219,"Bitvavo","Cryptocurrency",""]]},"BAT":{"Sell":[["2021-07-09T22:00:00.000Z","BAT","Sell",168.5,81.22,0.4820178041543027,"Binance","Cryptocurrency",""]],"Buy":[["2021-05-30T22:00:00.000Z","BAT","Buy",168.5,100,0.5934718100890207,"Binance","Cryptocurrency",""]]},"BNB":{"Sell":[["2021-07-09T22:00:00.000Z","BNB","Sell",0.7359,195.55,265.7290392716402,"Binance","Cryptocurrency",""]],"Buy":[["2021-05-18T22:00:00.000Z","BNB","Buy",0.7359,250,339.7200706617747,"Binance","Cryptocurrency",""]]},"LINK":{"Sell":[["2021-07-09T22:00:00.000Z","LINK","Sell",6.2,94.95,15.314516129032258,"Binance","Cryptocurrency",""]],"Buy":[["2021-07-04T22:00:00.000Z","LINK","Buy",6.2,261,42.096774193548384,"Binance","Cryptocurrency",""]]},"XMR":{"Sell":[["2021-07-09T22:00:00.000Z","XMR","Sell",3.09982,681,219.69017555858085,"Binance","Cryptocurrency",""]],"Buy":[["2021-05-24T22:00:00.000Z","XMR","Buy",1.51782,300,197.65189548167768,"Binance","Cryptocurrency",""],["2021-05-18T22:00:00.000Z","XMR","Buy",1.192,250,209.73154362416108,"Binance","Cryptocurrency",""],["2021-04-19T22:00:00.000Z","XMR","Buy",0.39,100,256.4102564102564,"Binance","Cryptocurrency",""]]},"RUNE":{"Buy":[["2021-07-04T22:00:00.000Z","RUNE","Buy",17.2,261,15.174418604651164,"Binance","Cryptocurrency",""]]},"SOL":{"Buy":[["2021-07-04T22:00:00.000Z","SOL","Buy",7.0007542,257,36.71033043839762,"Binance","Cryptocurrency",""]]},"KSM":{"Buy":[["2021-05-30T22:00:00.000Z","KSM","Buy",0.3869,100,258.46471956577926,"Binance","Cryptocurrency",""]]},"ADA":{"Buy":[["2021-05-18T22:00:00.000Z","ADA","Buy",164.44,250,1.5203113597664801,"Binance","Cryptocurrency",""],["2021-03-19T23:00:00.000Z","ADA","Buy",236,249.5,1.0572033898305084,"Binance","Cryptocurrency",""]]},"TSLA":{"Sell":[["2021-05-16T22:00:00.000Z","TSLA","Sell",1.8606349999999998,864,464.357598346801,"Trading212","Stock",""],["2021-01-18T23:00:00.000Z","TSLA","Sell",0.1449193,100.45,693.144391395763,"Trading212","Stock",""]],"Buy":[["2021-03-04T23:00:00.000Z","TSLA","Buy",0.3166797,150,473.6647154838153,"Trading212","Stock",""],["2021-03-02T23:00:00.000Z","TSLA","Buy",0.3841205,213.44,555.6589663920566,"Trading212","Stock",""],["2021-02-24T23:00:00.000Z","TSLA","Buy",0.3282356,188.45,574.1302893409489,"Trading212","Stock",""],["2021-02-22T23:00:00.000Z","TSLA","Buy",0.2252006,126.5,561.7214163727805,"Trading212","Stock",""],["2021-02-21T23:00:00.000Z","TSLA","Buy",0.2954647,184.99,626.0984814768059,"Trading212","Stock",""],["2021-02-21T23:00:00.000Z","TSLA","Buy",0.1624873,100,615.4327138182492,"Trading212","Stock",""],["2021-01-10T23:00:00.000Z","TSLA","Buy",0.1449193,100,690.03]
Produced errors
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: undefined is not iterable (cannot read property Symbol(Symbol.iterator))]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: undefined is not iterable (cannot read property Symbol(Symbol.iterator))]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
10:25:02 AM Info [TypeError: Cannot read property 'forEach' of undefined]
I am hoping someone here can help me solve this issue.
EDIT1: Adding the console.log output of tic.Sell
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Thu Oct 21 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'MATIC',
'Sell',
37.14,
61.82,
1.6645126548196014,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Wed Oct 20 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'TFUEL',
'Sell',
754,
182,
0.2413793103448276,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info [ [ Wed Oct 20 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'THETA',
'Sell',
81.4,
397.73,
4.886117936117936,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info undefined
2:02:36 PM Info undefined
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Sun Aug 08 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'safemars',
'Sell',
206130076,
16,
7.762089021885385e-8,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info [ [ Sun Aug 08 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'safemoon',
'Sell',
55800000,
90,
0.0000016129032258064516,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info [ [ Tue Jul 13 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'FRA:C36',
'Sell',
358,
343.68,
0.96,
'Bitvavo',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Jul 11 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.038237,
1116.81,
29207.57381593744,
'Binance',
'Cryptocurrency',
'' ],
[ Sun Jul 04 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.00537222,
257,
47838.69610700977,
'Binance',
'Cryptocurrency',
'' ],
[ Sun Jul 04 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.00543778,
261,
47997.528403135104,
'Binance',
'Cryptocurrency',
'' ],
[ Sun Jul 04 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.00533832,
261,
48891.78617992178,
'Binance',
'Cryptocurrency',
'' ],
[ Sun Jul 04 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.00550857,
261,
47380.71768172139,
'Binance',
'Cryptocurrency',
'' ],
[ Sun Jul 04 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.00008389000000000313,
0,
0,
'Bitvavo',
'Cryptocurrency',
'' ],
[ Sun Jun 06 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.038276,
1116.81,
29177.81377364406,
'Binance',
'Cryptocurrency',
'' ],
[ Sun May 02 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.01032703,
498.75,
48295.58934175654,
'Bitvavo',
'Cryptocurrency',
'' ],
[ Tue Apr 13 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BTC',
'Sell',
0.18413661999999997,
9757,
52987.83044893515,
'Bitvavo',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info [ [ Fri Jul 09 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BAT',
'Sell',
168.5,
81.22,
0.4820178041543027,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info [ [ Fri Jul 09 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'BNB',
'Sell',
0.7359,
195.55,
265.7290392716402,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info [ [ Fri Jul 09 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'LINK',
'Sell',
6.2,
94.95,
15.314516129032258,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info [ [ Fri Jul 09 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'XMR',
'Sell',
3.09982,
681,
219.69017555858085,
'Binance',
'Cryptocurrency',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info undefined
2:02:36 PM Info undefined
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Sun May 16 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'TSLA',
'Sell',
1.8606349999999998,
864,
464.357598346801,
'Trading212',
'Stock',
'' ],
[ Mon Jan 18 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'TSLA',
'Sell',
0.1449193,
100.45,
693.144391395763,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Apr 12 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'MSTR',
'Sell',
0.61234957,
413.5,
675.2678866092778,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Sun Apr 11 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'ACTC',
'Sell',
8,
109.91,
13.73875,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Apr 11 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'CTRM',
'Sell',
110,
43.46,
0.3950909090909091,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Apr 11 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'NVCN',
'Sell',
74,
61.85,
0.8358108108108109,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Apr 11 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'SFT',
'Sell',
13,
92.2,
7.092307692307693,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Sun Apr 11 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'SNGX',
'Sell',
65,
80.2,
1.2338461538461538,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Mar 28 2021 18:00:00 GMT-0400 (Eastern Daylight Time),
'LMND',
'Sell',
1.6704908,
121.86,
72.94862084843568,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Thu Mar 25 2021 19:00:00 GMT-0400 (Eastern Daylight Time),
'AZRX',
'Sell',
100,
112.35,
1.1235,
'Trading212',
'Stock',
'' ],
[ Thu Mar 25 2021 19:00:00 GMT-0400 (Eastern Daylight Time),
'AZRX',
'Sell',
59,
66.14,
1.1210169491525424,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Thu Mar 18 2021 19:00:00 GMT-0400 (Eastern Daylight Time),
'FB',
'Sell',
1,
244.81,
244.81,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Mar 15 2021 19:00:00 GMT-0400 (Eastern Daylight Time),
'ARB',
'Sell',
173.1368712,
482.54,
2.7870435491616994,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Wed Mar 10 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'GNUS',
'Sell',
72,
134.94,
1.8741666666666665,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Mar 08 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'ABNB',
'Sell',
0.4281417,
66.98,
156.44353259680148,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Mar 08 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'EQQQ',
'Sell',
1,
253.87,
253.87,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info undefined
2:02:36 PM Info undefined
2:02:36 PM Info [ [ Mon Mar 08 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'QDVA',
'Sell',
6,
48.93,
8.155,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Mar 08 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'SPY',
'Sell',
5,
273.66,
54.732000000000006,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Mar 08 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'SXLV',
'Sell',
7,
188.23,
26.889999999999997,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Mar 07 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'BABA',
'Sell',
1.7718482,
342.51,
193.3066275090609,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Mar 07 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'NIO',
'Sell',
1.888702,
61.79,
32.71558986012616,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Wed Feb 24 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'IQQH',
'Sell',
5,
60.09,
12.018,
'DeGiro',
'Stock',
'' ],
[ Wed Feb 24 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'IQQH',
'Sell',
2,
25.63,
12.815,
'DeGiro',
'Stock',
'' ],
[ Wed Feb 24 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'IQQH',
'Sell',
8,
102.46,
12.8075,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Feb 22 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'ESPO',
'Sell',
5,
191.18,
38.236000000000004,
'Trading212',
'Stock',
'' ],
[ Mon Feb 22 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'ESPO',
'Sell',
4.994859,
186.59,
37.35640986061869,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Feb 21 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'AMZN',
'Sell',
0.05568394,
147.39,
2646.903218414501,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Feb 21 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'COUP',
'Sell',
0.5194278,
155.58,
299.521896979715,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Feb 21 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'CRWD',
'Sell',
0.5152084,
99.28,
192.69872152705585,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Feb 21 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'FVRR',
'Sell',
0.5002109,
122.67,
245.2365592193213,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Feb 21 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'TTWO',
'Sell',
1,
160.04,
160.04,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Wed Jan 27 2021 18:00:00 GMT-0500 (Eastern Standard Time),
'NAKD',
'Sell',
55.421,
68.03,
1.2275130365745837,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Mon Dec 07 2020 18:00:00 GMT-0500 (Eastern Standard Time),
'OTGLY',
'Sell',
6.180493,
91.07,
14.735070487095445,
'Trading212',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Sun Nov 29 2020 18:00:00 GMT-0500 (Eastern Standard Time),
'WDI',
'Sell',
23,
8.21,
0.35695652173913045,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Tue Sep 08 2020 18:00:00 GMT-0400 (Eastern Daylight Time),
'ADP',
'Sell',
1,
114.63,
114.63,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Tue Sep 08 2020 18:00:00 GMT-0400 (Eastern Daylight Time),
'AQN',
'Sell',
3,
35.2,
11.733333333333334,
'DeGiro',
'Stock',
'' ] ]
2:02:36 PM Info [ [ Tue Sep 08 2020 18:00:00 GMT-0400 (Eastern Daylight Time),
'MGRC',
'Sell',
1,
53.1,
53.1,
'DeGiro',
'Stock',
'' ] ]
The problem exists inside the do/while loop.
Remainder becomes incredibly small (for MSTR, remainder = 5.551115123125783e-17 instead of 0) but not quite zero, so the loop runs again and an error is thrown. MSTR and THETA are thus never pushed into objects.
Fixed the issue by changing while (remainder > 1e-9). Might not be the best fix but it works.

How to convert JSON column to list in R

I have data set like this
genres
[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 14, "name": "Fantasy"}, {"id": 878, "name": "Science Fiction"}]
[{"id": 12, "name": "Adventure"}, {"id": 14, "name": "Fantasy"}, {"id": 28, "name": "Action"}]
[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 80, "name": "Crime"}]
[{"id": 28, "name": "Action"}, {"id": 80, "name": "Crime"}, {"id": 18, "name": "Drama"}, {"id": 53, "name": "Thriller"}]
[{"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}, {"id": 878, "name": "Science Fiction"}]
[{"id": 14, "name": "Fantasy"}, {"id": 28, "name": "Action"}, {"id": 12, "name": "Adventure"}]
[{"id": 16, "name": "Animation"}, {"id": 10751, "name": "Family"}]
Now I want to populate a drop down list of Shiny application so I want to convert the column JSON to data table, I use apply functions but does not get desired result, can someone please help me out.
Code:
lapply(dt, fromJSON(dt$genres))
I am new to R so it may be not best solution but here my finding which is working in my case, please let me know if any feedback
genre <- raw$genres
genreList <- lapply(genre,function(x) fromJSON(x))
genreList <- genreList[sapply(genreList, function(x) as.numeric(dim(x)[1])) > 0]
genreList <- genreList[!sapply(genreList, is.null)]
finalGenre <- unique(Reduce(function(...) merge(..., all=T), genreList))
And the output
id name
1 28 Action
2 53 Thriller
3 10769 Foreign
4 12 Adventure
5 10751 Family
6 878 Science Fiction
7 27 Horror
8 16 Animation
9 80 Crime
10 35 Comedy
11 10770 TV Movie
12 18 Drama
13 99 Documentary
14 10752 War
15 10402 Music
16 10749 Romance
17 14 Fantasy
18 37 Western
19 36 History
20 9648 Mystery

Parsing json files

Below is the code for visual analysis of a set of tweets obtained in a .json file. Upon interpreting , an error is shown at the map() function. Any way to fix it?
import json
import pandas as pd
import matplotlib.pyplot as plt
tweets_data_path = 'import_requests.txt'
tweets_data = []
tweets_file = open(tweets_data_path, "r")
for line in tweets_file:
try:
tweet = json.loads(line)
tweets_data.append(tweet)
except:
continue
print(len(tweets_data))
tweets = pd.DataFrame()
tweets['text'] = map(lambda tweet: tweet['text'], tweets_data)
These are the lines leading up to the 'ValueError' message I am getting for the above code :
Traceback (most recent call last):
File "tweet_len.py", line 21, in
tweets['text'] = map(lambda tweet: tweet['text'], tweets_data)
File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 1887, in setitem
self._set_item(key, value)
File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 1966, in _set_item
self._ensure_valid_index(value)
File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 1943, in _ensure_valid_index
raise ValueError('Cannot set a frame with no defined index '
ValueError: Cannot set a frame with no defined index and a value that cannot be converted to a Series
I am using Python3.
EDIT : Below is a sample of the twitter data collected ( .json format).
{
"created_at": "Sat Mar 05 05:47:23 +0000 2016",
"id": 705993088574033920,
"id_str": "705993088574033920",
"text": "Tumi Inc. civil war: Staff manning US ceasefire hotline 'can't speak Arabic' #fakeheadlinebot #learntocode #makeatwitterbot #javascript",
"source": "\u003ca href=\"http://javascriptiseasy.com\" rel=\"nofollow\"\u003eJavaScript is Easy\u003c/a\u003e",
"truncated": false,
"in_reply_to_status_id": null,
"in_reply_to_status_id_str": null,
"in_reply_to_user_id": null,
"in_reply_to_user_id_str": null,
"in_reply_to_screen_name": null,
"user": {
"id": 4382400263,
"id_str": "4382400263",
"name": "JavaScript is Easy",
"screen_name": "javascriptisez",
"location": "Your Console",
"url": "http://javascriptiseasy.com",
"description": "Get learning!",
"protected": false,
"verified": false,
"followers_count": 167,
"friends_count": 68,
"listed_count": 212,
"favourites_count": 11,
"statuses_count": 55501,
"created_at": "Sat Dec 05 11:18:00 +0000 2015",
"utc_offset": null,
"time_zone": null,
"geo_enabled": false,
"lang": "en",
"contributors_enabled": false,
"is_translator": false,
"profile_background_color": "000000",
"profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
"profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
"profile_background_tile": false,
"profile_link_color": "FFCC4D",
"profile_sidebar_border_color": "000000",
"profile_sidebar_fill_color": "000000",
"profile_text_color": "000000",
"profile_use_background_image": false,
"profile_image_url": "http://pbs.twimg.com/profile_images/673099606348070912/xNxp4zOt_normal.jpg",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/673099606348070912/xNxp4zOt_normal.jpg",
"profile_banner_url": "https://pbs.twimg.com/profile_banners/4382400263/1449314370",
"default_profile": false,
"default_profile_image": false,
"following": null,
"follow_request_sent": null,
"notifications": null
},
"geo": null,
"coordinates": null,
"place": null,
"contributors": null,
"is_quote_status": false,
"retweet_count": 0,
"favorite_count": 0,
"entities": {
"hashtags": [{
"text": "fakeheadlinebot",
"indices": [77, 93]
}, {
"text": "learntocode",
"indices": [94, 106]
}, {
"text": "makeatwitterbot",
"indices": [107, 123]
}, {
"text": "javascript",
"indices": [124, 135]
}],
"urls": [],
"user_mentions": [],
"symbols": []
},
"favorited": false,
"retweeted": false,
"filter_level": "low",
"lang": "en",
"timestamp_ms": "1457156843690"
}
I think you can use read_json:
import pandas as pd
df = pd.read_json('file.json')
print df.head()
contributors coordinates created_at entities \
contributors_enabled NaN NaN 2016-03-05 05:47:23 NaN
created_at NaN NaN 2016-03-05 05:47:23 NaN
default_profile NaN NaN 2016-03-05 05:47:23 NaN
default_profile_image NaN NaN 2016-03-05 05:47:23 NaN
description NaN NaN 2016-03-05 05:47:23 NaN
favorite_count favorited filter_level geo \
contributors_enabled 0 False low NaN
created_at 0 False low NaN
default_profile 0 False low NaN
default_profile_image 0 False low NaN
description 0 False low NaN
id id_str \
contributors_enabled 705993088574033920 705993088574033920
created_at 705993088574033920 705993088574033920
default_profile 705993088574033920 705993088574033920
default_profile_image 705993088574033920 705993088574033920
description 705993088574033920 705993088574033920
... is_quote_status lang \
contributors_enabled ... False en
created_at ... False en
default_profile ... False en
default_profile_image ... False en
description ... False en
place retweet_count retweeted \
contributors_enabled NaN 0 False
created_at NaN 0 False
default_profile NaN 0 False
default_profile_image NaN 0 False
description NaN 0 False
source \
contributors_enabled <a href="http://javascriptiseasy.com" rel="nof...
created_at <a href="http://javascriptiseasy.com" rel="nof...
default_profile <a href="http://javascriptiseasy.com" rel="nof...
default_profile_image <a href="http://javascriptiseasy.com" rel="nof...
description <a href="http://javascriptiseasy.com" rel="nof...
text \
contributors_enabled Tumi Inc. civil war: Staff manning US ceasefir...
created_at Tumi Inc. civil war: Staff manning US ceasefir...
default_profile Tumi Inc. civil war: Staff manning US ceasefir...
default_profile_image Tumi Inc. civil war: Staff manning US ceasefir...
description Tumi Inc. civil war: Staff manning US ceasefir...
timestamp_ms truncated \
contributors_enabled 2016-03-05 05:47:23.690 False
created_at 2016-03-05 05:47:23.690 False
default_profile 2016-03-05 05:47:23.690 False
default_profile_image 2016-03-05 05:47:23.690 False
description 2016-03-05 05:47:23.690 False
user
contributors_enabled False
created_at Sat Dec 05 11:18:00 +0000 2015
default_profile False
default_profile_image False
description Get learning!
[5 rows x 25 columns]

converting R dataframes to json object

Say I have the following dataframes:
df1 <- data.frame(Name = c("Harry","George"), color=c("#EA0001", "#EEEEEE"))
Name color
1 Harry #EA0001
2 George #EEEEEE
df.details <- data.frame(Name = c(rep("Harry",each=3), rep("George", each=3)),
age=21:23,
total=c(14,19,24,1,9,4)
)
Name age total
1 Harry 21 14
2 Harry 22 19
3 Harry 23 24
4 George 21 1
5 George 22 9
6 George 23 4
I know how to convert each df to json like this:
library(jsonlite)
toJSON(df.details)
[{"Name":"Harry","age":21,"total":14},{"Name":"Harry","age":22,"total":19},{"Name":"Harry","age":23,"total":24},{"Name":"George","age":21,"total":1},{"Name":"George","age":22,"total":9},{"Name":"George","age":23,"total":4}]
However, I am looking to get the following structure to my JSON data:
{
"myjsondata": [
{
"Name": "Harry",
"color": "#EA0001",
"details": [
{
"age": 21,
"total": 14
},
{
"age": 22,
"total": 19
},
{
"age": 23,
"total": 24
}
]
},
{
"Name": "George",
"color": "#EEEEEE",
"details": [
{
"age": 21,
"total": 1
},
{
"age": 22,
"total": 9
},
{
"age": 23,
"total": 4
}
]
}
]
}
I think the answer may be in how I store the data in a list in R before converting, but not sure.
Try this format:
df1$details <- split(df.details[-1], df.details$Name)[df1$Name]
df1
# Name color details
#1 Harry #EA0001 21, 22, 23, 14, 19, 24
#2 George #EEEEEE 21, 22, 23, 1, 9, 4
toJSON(df1)
#[{
#"Name":"Harry",
#"color":"#EA0001",
#"details":[
# {"age":21,"total":14},
# {"age":22,"total":19},
# {"age":23,"total":24}]},
#{
#"Name":"George",
#"color":"#EEEEEE",
#"details":[
# {"age":21,"total":1},
# {"age":22,"total":9},
# {"age":23,"total":4}]}
#]

Constructing request payload in R using rjson/jsonlite

My current code as seen below attempts to construct a request payload (body), but isn't giving me the desired result.
library(df2json)
library(rjson)
y = rjson::fromJSON((df2json::df2json(dataframe)))
globalparam = ""
req = list(
Inputs = list(
input1 = y
)
,GlobalParameters = paste("{",globalparam,"}",sep="")#globalparam
)
body = enc2utf8((rjson::toJSON(req)))
body currently turns out to be
{
"Inputs": {
"input1": [
{
"X": 7,
"Y": 5,
"month": "mar",
"day": "fri",
"FFMC": 86.2,
"DMC": 26.2,
"DC": 94.3,
"ISI": 5.1,
"temp": 8.2,
"RH": 51,
"wind": 6.7,
"rain": 0,
"area": 0
}
]
},
"GlobalParameters": "{}"
}
However, I need it to look like this:
{
"Inputs": {
"input1": [
{
"X": 7,
"Y": 5,
"month": "mar",
"day": "fri",
"FFMC": 86.2,
"DMC": 26.2,
"DC": 94.3,
"ISI": 5.1,
"temp": 8.2,
"RH": 51,
"wind": 6.7,
"rain": 0,
"area": 0
}
]
},
"GlobalParameters": {}
}
So basically global parameters have to be {}, but not hardcoded. It seemed like a fairly simple problem, but I couldn't fix it. Please help!
EDIT:
This is the dataframe
X Y month day FFMC DMC DC ISI temp RH wind rain area
1 7 5 mar fri 86.2 26.2 94.3 5.1 8.2 51 6.7 0.0 0
2 7 4 oct tue 90.6 35.4 669.1 6.7 18.0 33 0.9 0.0 0
3 7 4 oct sat 90.6 43.7 686.9 6.7 14.6 33 1.3 0.0 0
4 8 6 mar fri 91.7 33.3 77.5 9.0 8.3 97 4.0 0.2 0
This is an example of another data frame
> a = data.frame("col1" = c(81, 81, 81, 81), "col2" = c(72, 69, 79, 84))
Using this sample data
dd<-read.table(text=" X Y month day FFMC DMC DC ISI temp RH wind rain area
1 7 5 mar fri 86.2 26.2 94.3 5.1 8.2 51 6.7 0.0 0", header=T)
You can do
globalparam = setNames(list(), character(0))
req = list(
Inputs = list(
input1 = dd
)
,GlobalParameters = globalparam
)
body = enc2utf8((rjson::toJSON(req)))
Note that globalparam looks a bit funny because we need to force it to a named list for rjson to treat it properly. We only have to do this when it's empty.