How to split very large JSON file in R? - json

I am currently working with Amazon User Metadata of size roughly around 9 GB. I have this data in .json format and want to read in R, due to such huge volume of data I want to divide the files in small chunks (either by size or by number of elements). If you can help me out how can I do this in R. The format of file is as follows:
{
"asin": "0000031852",
"title": "Girls Ballet Tutu Zebra Hot Pink",
"price": 3.17,
"imUrl": "http://ecx.images-amazon.com/images/I/51fAmVkTbyL._SY300_.jpg",
"related":
{
"also_bought": ["B00JHONN1S", "B002BZX8Z6", "B00D2K1M3O", "0000031909", "B00613WDTQ", "B00D0WDS9A", "B00D0GCI8S", "0000031895", "B003AVKOP2", "B003AVEU6G", "B003IEDM9Q", "B002R0FA24", "B00D23MC6W", "B00D2K0PA0", "B00538F5OK", "B00CEV86I6", "B002R0FABA", "B00D10CLVW", "B003AVNY6I", "B002GZGI4E", "B001T9NUFS", "B002R0F7FE", "B00E1YRI4C", "B008UBQZKU", "B00D103F8U", "B007R2RM8W"],
"also_viewed": ["B002BZX8Z6", "B00JHONN1S", "B008F0SU0Y", "B00D23MC6W", "B00AFDOPDA", "B00E1YRI4C", "B002GZGI4E", "B003AVKOP2", "B00D9C1WBM", "B00CEV8366", "B00CEUX0D8", "B0079ME3KU", "B00CEUWY8K", "B004FOEEHC", "0000031895", "B00BC4GY9Y", "B003XRKA7A", "B00K18LKX2", "B00EM7KAG6", "B00AMQ17JA", "B00D9C32NI", "B002C3Y6WG", "B00JLL4L5Y", "B003AVNY6I", "B008UBQZKU", "B00D0WDS9A", "B00613WDTQ", "B00538F5OK", "B005C4Y4F6", "B004LHZ1NY", "B00CPHX76U", "B00CEUWUZC", "B00IJVASUE", "B00GOR07RE", "B00J2GTM0W", "B00JHNSNSM", "B003IEDM9Q", "B00CYBU84G", "B008VV8NSQ", "B00CYBULSO", "B00I2UHSZA", "B005F50FXC", "B007LCQI3S", "B00DP68AVW", "B009RXWNSI", "B003AVEU6G", "B00HSOJB9M", "B00EHAGZNA", "B0046W9T8C", "B00E79VW6Q", "B00D10CLVW", "B00B0AVO54", "B00E95LC8Q", "B00GOR92SO", "B007ZN5Y56", "B00AL2569W", "B00B608000", "B008F0SMUC", "B00BFXLZ8M"],
"bought_together": ["B002BZX8Z6"]
},
"salesRank": {"Toys & Games": 211836},
"brand": "Coxlures",
"categories": [["Sports & Outdoors", "Other Sports", "Dance"]]
}

Related

AWS DynamoDB Issues adding values to existing table

I have already created a table called Sensors and identified Sensor as the hash key. I am trying to add to the table with my .json file. The items in my file look like this:
{
"Sensor": "Room Sensor",
"SensorDescription": "Turns on lights when person walks into room",
"ImageFile": "rmSensor.jpeg",
"SampleRate": "1000",
"Locations": "Baltimore, MD"
}
{
"Sensor": "Front Porch Sensor",
"SensorDescription": " ",
"ImageFile": "fpSensor.jpeg",
"SampleRate": "2000",
"Locations": "Los Angeles, CA"
}
There's 20 different sensors in the file. I was using the following command:
aws dynamodb batch-write-item \
--table-name Sensors \
--request-items file://sensorList.json \
--returned-consumed-capacity TOTAL
I get the following error:
Error parsing parameter '--request-items': Invalid JSON: Extra data: line 9 column 1 (char 189)
I've tried adding --table name Sensors to the cl and it says Unknown options: --table-name, Sensors. I've tried put-item and a few others. I am trying to understand what my errors are, what I need to change in my .json if anything, and what I need to change in my cl. Thanks!

Your input file is not a valid json. You are missing a comma to separate both objects, and you need to enclose everything with brackets [ ..... ]

Altering JSON Structure

So I am using a webscraper to pull information on sneakers from a website. The son data that comes back is structured like so
[
{
"web-scraper-order": "1554084909-97",
"web-scraper-start-url": "https://www.goat.com/sneakers",
"productlink": "$200AIR JORDAN 6 RETRO 'INFRARED' 2019",
"productlink-href": "https://www.goat.com/sneakers/air-jordan-6-retro-black-infrared-384664-060",
"name": "Air Jordan 6 Retro 'Infrared' 2019",
"price": "Buy New - $200",
"description": "The 2019 edition of the Air Jordan 6 Retro ‘Infrared’ is true to the original colorway, which Michael Jordan wore when he captured his first NBA title. Dressed primarily in black nubuck with a reflective 3M layer underneath, the mid-top features Infrared accents on the midsole, heel tab and lace lock. Nike Air branding adorns the heel and sockliner, an OG detail last seen on the 2000 retro.",
"releasedate": "2019-02-16",
"colorway": "Black/Infrared 23-Black",
"brand": "Air Jordan",
"designer": "Tinker Hatfield",
"technology": "Air",
"maincolor": "Black",
"silhouette": "Air Jordan 6",
"nickname": "Infrared",
"category": "lifestyle",
"image-src": "https://image.goat.com/crop/1250/attachments/product_template_additional_pictures/images/018/675/318/original/464372_01.jpg.jpeg"
},
{
"web-scraper-order": "1554084922-147",
"web-scraper-start-url": "https://www.goat.com/sneakers",
"productlink": "$190YEEZY BOOST 350 V2 'CREAM WHITE / TRIPLE WHITE'",
"productlink-href": "https://www.goat.com/sneakers/yeezy-boost-350-v2-cream-white-cp9366",
"name": "Yeezy Boost 350 V2 'Cream White / Triple White'",
"price": "Buy New - $220",
"description": "First released on April 29, 2017, the Yeezy Boost 350 V2 ‘Cream White’ combines a cream Primeknit upper with tonal cream SPLY 350 branding, and a translucent white midsole housing full-length Boost. Released again in October 2018, this retro helped fulfill Kanye West’s oft-repeated ‘YEEZYs for everyone’ Twitter mantra, as adidas organized the biggest drop in Yeezy history by promising pre-sale to anyone who signed up on the website. Similar to the first release, the ‘Triple White’ 2018 model features a Primeknit upper, a Boost midsole and custom adidas and Yeezy co-branding on the insole.",
"releasedate": "2017-04-29",
"colorway": "Cream White/Cream White/Core White",
"brand": "adidas",
"designer": "Kanye West",
"technology": "Boost",
"maincolor": "White",
"silhouette": "Yeezy Boost 350",
"nickname": "Cream White / Triple White",
"category": "lifestyle",
"image-src": "https://image.goat.com/crop/1250/attachments/product_template_additional_pictures/images/014/822/695/original/116662_03.jpg.jpeg"
},
However, I want to change it so that the top level node is sneakers and the next level down would be a specific sneaker brand ( Jordan, Nike, Adidas) and then the list of sneakers that belong to that brand. So my JSON structure would look something like this
Sneakers {
Adidas :{
[shoe1,
shoe2,
....
] },
Jordan: {
[shoe1,
shoe2,
....
]
}
}
I am not sure what tool I could use to do that. Any help would be greatly appreciated. All I have at the moment is the JSON file and it is not in the structure that I want it to be in.

One way of doing this would be to populate a dict whose keys are brand names and their values are lists of sneaker records. Assuming that data is your original list, here's the code:
sneakers_by_brand = {}
for record in data:
if sneakers_by_brand.get(record.get("brand")):
sneakers_by_brand[record.get("brand")].append(record)
else:
sneakers_by_brand[record.get("brand")] = [record]
print(sneakers_by_brand)

Error in fromJSON in R

I am trying to convert JSON file into data frame in R and had this error when I ran fromJSON in 'jsonlite' package.
Error in fromJSON(content, handler, default.size, depth, allowComments, :
Invalid JSON Node
I figured that the variable order are different in each row. In specific, for example, in the first row, "categories" is the second variable whereas it is the last variable in the third row (Please take a look at the data below). This is how the original JSON file is formatted, and I have several hundred thousand rows like this so can't manually change it.
Anyone can suggest how to fix this and successfully convert the JSON into data frame in R?
[{"asin": "0188399313", "categories": [["Baby"]], "description": "Wee-Go Glass baby bottles by LifeFactory (Babylife) are designed to grow with your child. The included clear cover can also serve as an easy to hold cup. Twist on the solid cap (sold separately) and use your bottles for storing juice or snacks. Perfect for a lunchbox or traveling. The bright colored silicone sleeve (patent pending) helps to protect the bottle from breakage and provides a great gripping surface and tactile experience during feeding. The bottle and sleeve can be boiled or put in the dishwasher together. They can also go in the freezer, making breast milk storage simple.", "title": "Lifefactory 4oz BPA Free Glass Baby Bottles - 4-pack-raspberry and Lilac", "price": 69.99, "imUrl": "http://ecx.images-amazon.com/images/I/41SwthpdD9L._SX300_.jpg", "brand": "Lifefactory", "related": {"also_bought": ["B002SG7K7A", "B003CJSXW8", "B004PW4186", "B002O3JH9Q", "B002O3NLIO", "B004HGSU28"], "also_viewed": ["B003CJSXW8", "B0052QOL1Q", "B004PW4186", "B00EN0OLZ8", "B00EN0OOQY", "B0049YS46K", "B00E64CBLM", "B00F9YOOS6", "B00AH9RPVQ", "B00BCU2R7G", "B002O3NLIO", "B008NZ4X2K", "B005NIDFEW", "B00DKPJCH4", "B00CZNGWWK", "B00DAKJIQ4", "B005CT55IQ", "B0049YRJM0", "B0071IEWD0", "B00E64CA68", "B00IUB3SKK", "B00A7AA6XY", "B001F50FFE", "B002HU9EO4", "B007HP11SQ", "B009WPUMX4", "B002O3JH9Q", "B00F2FT3K6", "B00I5CR35A", "B00BCTY5EK", "B002SG7K7A", "B00F2FLU2U", "B0062ZK0GQ", "B002UOFR66", "B0055LKQQ2", "B00A0FGN8I", "B00HMYCG2W", "B00DHFLUO0", "B0040HMPA2", "B00I5CT9XE", "B008B5MMNO", "B00BQYVNGO", "B00925WM28", "B00BGKC3EY", "B005Q3LSDO", "B0038JDVCE", "B0045I6IA4"], "bought_together": ["B002SG7K7A", "B003CJSXW8"], "buy_after_viewing": ["B003CJSXW8", "B0052QOL1Q", "B004PW4186", "B002SG7K7A"]},},
{"asin": "0188399518", "categories": [["Baby"]], "description": "The Planet Wise Flannel Wipes are 10 super soft, color coordinated, high-quality wipes that fit perfectly into the Planet Wise Wipe PouchTM.", "title": "Planetwise Flannel Wipes", "price": 15.95, "imUrl": "http://ecx.images-amazon.com/images/I/41otjnA4OGL._SY300_.jpg", "brand": "Planet Wise", "related": {"also_bought": ["B00G96N3YY", "B003XSEV2O", "B000138GNY", "B005WWIE3G", "B005WWI0DA", "B005WWIMGA", "B00DS4WPNK", "B0039VCRPI", "B001QIN6ME", "B004GMGLN8", "B00CJ2OWUG", "B004BDNJW8", "B003N0JXSO", "B003X3R6TO", "B002T5Q01C", "B006J2U4T0", "B00JBJDEC2", "B00380LVLG", "B00GJVM2NW", "B003AJHDQW", "B0043EDGP0", "B0012IJBUE", "B001OI0YWG", "B005DL5970", "B00305GSKS", "B00CQ9UUK8", "B002Y27PQ4", "B00AH8J448", "B00A3JXVZY", "B00483GAJU", "B001J6O6B8", "B0019ID6G2", "B005DL7LGM", "B00APSYDXM", "B004D5KJJA", "B0021HR94K", "0757302661", "B001HX4DNE"], "also_viewed": ["B000138GNY", "B00G96N3YY", "B006J2U4T0", "B00GLBR3C0", "B0039VCRPI", "B0012IJBUE", "B001IA2XGK", "B001R2L6PS", "B003X3R6TO", "B00DS4WPNK", "B005WWIE3G", "B004BDNJW8", "B005WWIMGA", "B005WWI0DA", "B003XSEV2O", "B004A5VGPE", "B004A5TTGC", "B00JBJDEC2", "B004GMGLN8", "B009W6PM9M", "B004RGL9I0", "B0040IWMTK", "B00J3C0XC6", "B00HO1HBC8", "B00G96N18M", "B00A3V0XDA", "B0032AMM9M", "B001QIN6ME", "B00A3V0XEY", "B00CIUAB40", "B00J3C0Y1G", "B005IPE0KI", "B00L0W5B7Y", "B00FZGLKQW", "B00HBYOE3W", "B004A5TT4O", "B00B6KS578", "B002HOQOMA", "B004G792KW", "B002UD6BZS", "B002XVRKHK", "B002HOQOUW", "B002UD6C1Q", "B00APSYDXM", "B00H07AZ42", "B002LZX2BG", "B00BFQVZNO", "B00EZKNVYS", "B00DCLYYJW", "B00AHG8H1Q", "B002QZ64T8", "B003N0JXSO", "B003AJHDQW", "B00DN9NW32", "B00CHPJ0I4", "B005WXMJUY", "B00D4LFC3W"], "bought_together": ["B00G96N3YY", "B003XSEV2O"], "buy_after_viewing": ["B000138GNY", "B006J2U4T0", "B00G96N3YY", "B005WWIMGA"]},},
{"asin": "0188399399", "description": "The Planet Wise Wipe PouchTM features our patent pending no-leak design so your wipes will stay moist and not wick into your diaper bag. Features a unique snap down design to eliminate wicking from the zipper area. Most standard size wipes will fit into our Wipe PouchTM perfectly! Pouches are made of the same high quality fabrics as our Wet Bags, this pouch will last and last. Pouches are anti-microbial too! After you are done with the baby years, keep this pouch to use for just about anything you need to transport: make-up, toiletries, anything. Made in the USA!", "title": "Planetwise Wipe Pouch", "price": 10.95, "imUrl": "http://ecx.images-amazon.com/images/I/61x8h9u6mxL._SY300_.jpg", "related": {"also_bought": ["B005WWI0DA", "B005WWIMGA", "B006J2U4T0", "B000138GNY", "B003XSEV2O", "B005WWIE3G", "B00DS4WPNK", "B0039VCRPI", "B003GSLCOG", "B0018B15FE", "B002MN3JY2"], "also_viewed": ["B005WWIMGA", "B00G96N3YY", "B00DAI76TC", "B0067GKHVS", "B005WWIE3G", "B004RGL9I0", "B000138GNY", "B00JBJDEC2", "B00CMCJ2AS", "B005WWI0DA", "B002UD6C16", "B00E401FVU", "B004GMGLN8", "B0012IJBUE", "B00BFQVZNO", "B006J2U4T0", "B000VV21G4", "B002HOQOUW", "B003X3R6TO", "B000GZJIVQ", "B0052AIF00", "B005GQ6606", "B003TZHCA4", "B001QIN6ME", "B003XSEV2O", "B00GLBR3C0", "B00DW9R03G", "B00J3C0XC6", "0188399542", "B00DQL6CIE", "B002UD6C1G", "B00A3V0XDA", "B0039VCRPI", "B00EZKNVYS", "B006QRFDHQ", "B00DS4WPNK", "B00CQ9UUK8", "B0087A2PKS", "B00GGNICSW", "B0037NXP18", "B000YCNLSC", "B00FJG945W", "B00FZGLKQW", "B004A5VGPE", "B00D4LFC3W", "B00G96N18M", "B00439UNR4", "B004A5TT4O", "B005DL7LGM", "B006SD2JUW", "B006Z6HXYY", "B006QRFDZI", "B00I0P7VYK", "B00AZWDM0I", "B000ZKHVMU", "B007799PGC", "B00H58UNMU", "B00JKUU3TE"], "buy_after_viewing": ["B005WWIMGA", "B00G96N3YY", "B00DAI76TC", "B0067GKHVS"]},, "categories": [["Baby"]]},
{"asin": "0316967297", "description": "Hand crafted set includes 1 full quilt (76x86 inches) and 2 standard shams (20x26 inches). Face cloth and fill are 100 natural cotton. Prewashed for out of the bag comfort. Hand crafted with embroidery. Machine washable. Made in China", "title": "Annas Dream Full Quilt with 2 Shams", "price": 109.95, "imUrl": "http://ecx.images-amazon.com/images/I/51%2BZ1%2BNeukL._SY300_.jpg", "related": {"also_viewed": ["B009LTER3W", "B00575TI5Q", "B004NSYYJI"], "buy_after_viewing": ["B009LTER3W", "B001MX5EE6", "B00575TI5Q", "B0029009TG"]},, "categories": [["Baby"]]},
{"asin": "0615447279", "categories": [["Baby"]], "description": "Thumbuddy To Love- The Binky Fairy helps children give up pacifier sucking without the fuss and tears. The adorable book comes with a matching Binky Fairy puppet and just like the Tooth Fairy, the book is read the night before so children understand where thier pacifiers will go. They awake to find an adorable Binky Fairy puppet under thier pillow knowing that the Binky Fairy came! Each book comes with a success chart and stickers. Winner of the PTPA Awards (Parent Tested, Parent Approved). Recommended for ages 2-4.", "title": "Stop Pacifier Sucking without tears with Thumbuddy To Love s Binky Fairy Puppet and Adorable Book", "price": 16.95, "imUrl": "http://ecx.images-amazon.com/images/I/51RKKENlq%2BL._SY300_.jpg", "brand": "", "related": {"also_bought": ["0979670004", "1601310234", "B005G172KE", "1575422573", "1905417896", "B0044D0HA2", "B001GQ2CPI"], "also_viewed": ["0979670004", "1575422573", "1493535943", "0615273645", "1601310234", "0992616727", "0615273793", "0763623644", "1907152962", "0992616719", "B0016ZX7AI", "0698400488", "1608442713", "0451416058", "1581176848", "B0072CTECY", "B005G172KE", "B0016P8K3W", "0375822704", "0979201004", "0961478020", "160131048X", "1572245859"], "bought_together": ["B005G172KE"], "buy_after_viewing": ["0979670004", "1575422573", "B005G172KE", "1493535943"]},}]

how can i read a custom DataSource(*.json) by Cesium

i have a grb2 file, i use grib2json covert it to .json, here is the json code:
[
{
"header":{
"discipline":0,
"disciplineName":"Meteorological products",
"gribEdition":2,
"gribLength":455039,
"center":7,
"centerName":"US National Weather Service - NCEP(WMC)",
"subcenter":0,
"refTime":"2016-10-05T12:00:00.000Z",
"significanceOfRT":1,
"significanceOfRTName":"Start of forecast",
"productStatus":0,
"productStatusName":"Operational products",
"productType":1,
"productTypeName":"Forecast products",
"productDefinitionTemplate":0,
"productDefinitionTemplateName":"Analysis/forecast at horizontal level/layer at a point in time",
"parameterCategory":2,
"parameterCategoryName":"Momentum",
"parameterNumber":3,
"parameterNumberName":"V-component_of_wind",
"parameterUnit":"m.s-1",
"genProcessType":2,
"genProcessTypeName":"Forecast",
"forecastTime":0,
"surface1Type":103,
"surface1TypeName":"Specified height level above ground",
"surface1Value":10.0,
"surface2Type":255,
"surface2TypeName":"Missing",
"surface2Value":0.0,
"gridDefinitionTemplate":0,
"gridDefinitionTemplateName":"Latitude_Longitude",
"numberPoints":259920,
"shape":6,
"shapeName":"Earth spherical with radius of 6,371,229.0 m",
"gridUnits":"degrees",
"resolution":48,
"winds":"true",
"scanMode":0,
"nx":720,
"ny":361,
"basicAngle":0,
"subDivisions":0,
"lo1":0.0,
"la1":90.0,
"lo2":359.5,
"la2":-90.0,
"dx":0.5,
"dy":0.5
},
"data":[
3.92,
3.89,
......
-3.06,
-3.07
]
}
]
when i use Cesium1.26 to read the json file, it works, but i can't get the header and data, can anyone help me about this problem?

Importing JSON file into Firebase error

I keep getting that there is an error uploading/importing my JSON file into Firebase. I initially had an excel spreadsheet that I saved as a CSV file, then I used a CSV to JSON converter.
I validated the JSON file (which have the .json extension) with a couple of online tools.
Though, I'm still getting an error.
Here is an example of my JSON:
{
"Rk": 1,
"Tm": "SEA",
"H/A": "H",
"DOW": "Sun",
"Opp": "CLE",
"QB": "Russell Wilson",
"Grade": "BLUE",
"Def mu pts": 4,
"Inj status": 0,
"Notes": "Got to wonder if not having a proven power RB under center will negatively impact Wilson's production.",
"TFS $50K": "$8,300",
"Init sal": "$8,300",
"Var": "$0",
"WC": 0
}

The issue is your key's..
Firebase keys must be:
UTF-8 encoded, cannot contain . $ # [ ] / or ASCII control characters
0-31 or 127
your $50k key and the H/A are the issues.