Dears,
Can someone help me out reading the content of JSON file in LINUX machine without using JQ, Python & Ruby. Looking for purely SHELL scripting. We need to iterate the values if multiple records found. In the below case we have 2 set of records, which needs to iterated.
{
"version": [
"sessionrestore",1
],
"windows": [
{
"tabs": [
{
"entries": [
{
"url": "http://orf.at/#/stories/2.../" (http://orf.at/#/stories/2.../%27) ,
"title": "news.ORF.at",
"charset": "UTF-8",
"ID": 9588,
"docshellID": 298,
"docIdentifier": 10062,
"persist": true
},
{
"url": "http://oracle.at/#/stories/2.../" (http://oracle.at/#/stories/2.../%27) ,
"title": "news.at",
"charset": "UTF-8",
"ID": 9589,
"docshellID": 288,
"docIdentifier": 00062,
"persist": false
}
]
}
}
}
Related
Background
I am using PowerShell 7.
Yesterday, I asked this question on help merging some JSON together and saving it, it worked great.
I have now ran into another problem that my new gained knowledge doesn't help with - escaped characters.
Problem
I have some JSON like so:
{
"toolcache": [
{
"name": "Python",
"url": "https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json",
"platform": "linux",
"platform_version": "18.04",
"arch": "x64",
"versions": [
"2.7.*",
"3.6.*",
"3.7.*",
"3.8.*",
"3.9.*",
"3.10.*"
]
}
],
"powershellModules": [
{"name": "MarkdownPS"},
{"name": "Microsoft.Graph"},
{"name": "Pester"},
{"name": "PSScriptAnalyzer"}
]
}
And I try to apply the knowledge in the other post:
### Add to JSON Toolet
# Try to escape characters as plain text using heredoc - Is this is the issue?
$ToolsetsToAdd = #"
{"name": "SqlServer"}
"#
$ToolsetFile = "toolset-2004.json"
$ToolsetObject = $(Get-Content $ToolsetFile -Raw)
$ToolSetSystemObject =$($ToolsetObject | ConvertFrom-Json)
$ToolSetSystemObject.powershellModules += $ToolsetsToAddJson
$ToolSetSystemObject | ConvertTo-Json -Depth 100 | Set-Content $ToolsetFile
And when I run it, I get:
{
"toolcache": [
{
"name": "Python",
"url": "https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json",
"platform": "linux",
"platform_version": "18.04",
"arch": "x64",
"versions": [
"2.7.*",
"3.6.*",
"3.7.*",
"3.8.*",
"3.9.*",
"3.10.*"
]
}
],
"powershellModules": [
{
"name": "MarkdownPS"
},
{
"name": "Microsoft.Graph"
},
{
"name": "Pester"
},
{
"name": "PSScriptAnalyzer"
},
"{\"name\": \"SqlServer\"}"
],
I have tried a variation of escapes using things like `"`" but couldn't figure this one out either.
This to my knowledge is expanded JSON, so I'm not sure its an error, but its not copying and escaping things like I expected. Whether this is avoidable is what I want to find out from someone who knows better!
What I want to do:
Ideally, I'd like a correctly formatted and parsed JSON file like so:
{
"toolcache": [
{
"name": "Python",
"url": "https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json",
"platform": "linux",
"platform_version": "18.04",
"arch": "x64",
"versions": [
"2.7.*",
"3.6.*",
"3.7.*",
"3.8.*",
"3.9.*",
"3.10.*"
]
}
],
"powershellModules": [
{"name": "MarkdownPS"},
{"name": "Microsoft.Graph"},
{"name": "Pester"},
{"name": "PSScriptAnalyzer"}
{"name": "SqlServer"}
]
}
You are treating the array in $ToolSetSystemObject.powershellModules as if it were an array of strings, but in fact this is an array of objects
To add a new object to it, just do
$ToolSetSystemObject.powershellModules += #{name = "SqlServer"}
or
$ToolSetSystemObject.powershellModules += [PsCustomObject]#{name = "SqlServer"}
So I have a JSON file I got from Postman which is returning as an empty object. This is how I'm reading it.
import regscooter from './json_files/reginald_griffin_scooter.json'
const scoot = regscooter;
const CustomerPage = () => {...}
reginald_griffin_scooter.json
{
"success": true,
"result": {
"id": "hhhhhhhhhh",
"model": "V1 Scooter",
"name": "hhhhhhhhhh",
"status": "active",
"availabilityStatus": "not-available",
"availabilityTrackingOn": true,
"serial": "hhhhhhhhhhhh",
"createdByUser": "hhhhhhhhK",
"createdByUsername": "hhhhhhhh",
"subAssets": [
"F0lOjWBAnG"
],
"parts": [
"hhhhhhhh"
],
"assignedCustomers": [
"hhhhhhhhh"
],
"createdAt": "2019-12-03T21:47:26.218Z",
"updatedAt": "2020-06-26T22:05:54.526Z",
"customFieldsAsset": [
{
"id": "hhhhhhh",
"name": "MAC",
"value": "hhhhhhhh",
"asset": "hhhhhhhhhh",
"user": "hhhhhhhhh",
"createdAt": "2019-12-03T21:47:26.342Z",
"updatedAt": "2019-12-11T16:29:24.732Z"
},
{
"id": "hhhhhhhh",
"name": "IMEI",
"value": "hhhhhhh",
"asset": "hhhhhhh",
"user": "hhhhhhhhhh",
"createdAt": "2019-12-03T21:47:26.342Z",
"updatedAt": "2019-12-11T16:29:24.834Z"
},
{
"id": "hhhhhhhhh",
"name": "Key Number",
"value": "NA",
"asset": "hhhhhhhhh",
"user": "hhhhhhhhhhh",
"createdAt": "2019-12-03T21:47:26.342Z",
"updatedAt": "2019-12-11T16:29:24.911Z"
}
]
}
}
The error is that "const scoot" is being shown as an empty object {}. I made sure to save a ton of times everywhere. I am able to read through the imported JSON file in other variables in similar ways, so I don't know why I can't parse this one. I just want to access the JSON object inside this. Also I omitted some information with hhhhh because of confidentiality.
EDIT: The code works, but it still has a red line beneath result when I do:
const scoot = regscooter.result.id;
It would be much more effective if you will provide an example in codesandbox or so.
However at first look it might be a parser issue ( maybe you are using Webpack with missing configuration for parsing files with json extension ), meaning we need more information to provide you with a full answer ( maybe solution ? ).
Have you tried to do the next:
const scoot = require('./json_files/reginald_griffin_scooter.json');
I'm writing a build.proj file to build and deploy a application. I have one application.configuration file which contains some project configurations. application.configuration is written in json format.
{
"Clients": [
{
"Id": "xyz",
"Name": "test",
"Flow": "2"
}
],
"Scopes": [
{
"Name": "roles",
"DisplayName": "roles",
"Claims": [
{
"Name": "role"
}
]
}
]
}
In build.proj file a want to read and update data from this file so what should i use?
Because MSBUILD Tasks only reads and update data from .xml files.
Writting one shell script to automatically get list of name, current and latest available version from raw json data.
I am trying to format JSON data stored in file using shell script. I tried using JQ command line JSON parser.
I want to get formatted JSON data in script. Their is advanced option provided in JQ for same scenario. I am not able to use it properly.
Example: File containing Following JSON
{
"endpoint": {
"name": "test-plugin",
"version": "0.0.1"
},
"dependencies": {
"plugin1": {
"main": {
"name": "plugin1name",
"description": "Dummy text"
},
"pkgMeta": {
"name": "plugin1name",
"version": "0.0.1"
},
"dependencies": {},
"versions": [
"0.0.5",
"0.0.4",
"0.0.3",
"0.0.2",
"0.0.1"
],
"update": {
"latest": "0.0.5"
}
},
"plugin2": {
"main": {
"name": "plugin2name",
"description": "Dummy text"
},
"pkgMeta": {
"name": "plugin2name",
"version": "0.1.1"
},
"dependencies": {},
"versions": [
"0.1.5",
"0.1.4",
"0.1.3",
"0.1.2",
"0.1.1"
],
"update": {
"latest": "0.1.5"
}
}
}
}
Trying to get result in format
[{name: "plugin1name",
c_version: "0.0.1",
n_version: "0.0.5"
},
{name: "plugin2name",
c_version: "0.1.1",
n_version: "0.1.5"}]
Can someone suggest anything ?
Your json file is not valid at: .dependencies.pkgMeta.version.
After fixing your json file, try this command:
jq '
.dependencies |
to_entries |
map(.value |
{
name: .main.name,
c_version: .pkgMeta.version,
n_version: .update.latest
}
)' input.json
The result is:
[
{
"name": "plugin1name",
"c_version": "0.0.1",
"n_version": "0.0.5"
},
{
"name": "plugin2name",
"c_version": "0.1.1",
"n_version": "0.1.5"
}
]
EDIT 2014-05-01: I tried fromJSON first (as suggested below), but that only parsed the first line. I found out that there were commas missing between the brackets of each JSON line so I changed that in TextEdit and saved the file. I also added [ at the beginning of the file and ] at the end and then it worked with JSON. Now the next step: from a list (with embedded lists) to a dataframe (or csv).
I get a data package from edX every now and then on the courses we are evaluating. Some of these are just plain .csv files which are quite easy to handle, others are more difficult for me (not having a CS or programming background).
I have 2 files I want to open and parse into csv files for analysis in R. I have tried many many json2csv tools out there, but to no avail. I also tried the simple methods described here to turn json into csv.
The data is confidential, so I cannot share the entire data set, but will share the first two lines of the file, maybe that helps. The problem is that nowhere I find anything about .mongo files, which to me seems quite strange, do they even exist? Or is this just a JSON file that may be corrupted (which could explain the errors)?
Any suggestions are welcome.
The first 2 lines in one of the .mongo files:
{
"_id": {
"$oid": "52d1e62c350e7a3156000009"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
],
"at_position_list": [
],
"body": "the delft university accredited course with the scholarship (fundamentals of water treatment) is supposed to start in about a month's time. But have the scholarship list been published? Any tentative date??",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"author_id": "269835",
"comment_thread_id": {
"$oid": "52cd40c5ab40cf347e00008d"
},
"author_username": "tachak59",
"sk": "52d1e62c350e7a3156000009",
"updated_at": {
"$date": 1389487660636
},
"created_at": {
"$date": 1389487660636
}
}{
"_id": {
"$oid": "52d0a66bcb3eee318d000012"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
{
"$oid": "52c63278100c07c0d1000028"
}
],
"at_position_list": [
],
"body": "I got it. Thank you!",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"parent_id": {
"$oid": "52c63278100c07c0d1000028"
},
"author_id": "2655027",
"comment_thread_id": {
"$oid": "52c4f303b03c4aba51000013"
},
"author_username": "dmoronta",
"sk": "52c63278100c07c0d1000028-52d0a66bcb3eee318d000012",
"updated_at": {
"$date": 1389405803386
},
"created_at": {
"$date": 1389405803386
}
}{
"_id": {
"$oid": "52ceea0cada002b72c000059"
},
"votes": {
"up": [
],
"down": [
],
"up_count": 0,
"down_count": 0,
"count": 0,
"point": 0
},
"visible": true,
"abuse_flaggers": [
],
"historical_abuse_flaggers": [
],
"parent_ids": [
{
"$oid": "5287e8d5906c42f5aa000013"
}
],
"at_position_list": [
],
"body": "if u please send by mail \n",
"course_id": "DelftX/CTB3365x/2013_Fall",
"_type": "Comment",
"endorsed": false,
"anonymous": false,
"anonymous_to_peers": false,
"parent_id": {
"$oid": "5287e8d5906c42f5aa000013"
},
"author_id": "2276302",
"comment_thread_id": {
"$oid": "528674d784179607d0000011"
},
"author_username": "totah1993",
"sk": "5287e8d5906c42f5aa000013-52ceea0cada002b72c000059",
"updated_at": {
"$date": 1389292044203
},
"created_at": {
"$date": 1389292044203
}
}
R doesn't have "native" support for these files but there is a JSON parser with the rjson package. So I might load my .mongo file with:
myfile <- "path/to/myfile.mongo"
myJSON <- readLines(myfile)
myNiceData <- fromJSON(myJSON)
Since RJson converts into a data structure that fits the object being read, you'll have to do some additional snooping but once you have an R data type you shouldn't have any trouble working with it from there.
Another package to consider when parsing JSON data is jsonlite. It will make data frames for you so you can write them to a csv format with write.table or some other applicable method for writing objects.
NOTE: if it is easier to connect to the MongoDB and get the data from a request, then RMongo may be a good bet. The R-Bloggers also made a post about using RMongo that has a nice little walkthrough.
I used RJSON as suggested by #theWanderer and with the help of a colleague wrote the following code to parse the data into columns, choosing the specific columns that are needed, and checking each of the instances if they return the right variables.
Entire workflow:
Checked some of the data in jsonlint - corrected the errors → },{ instead of }{ between each line and [ and ] at the beginning and end of the file
Made a smaller file to play with, containing about 11 JSON lines
Used the code below to parse the datafile - however, checking the different listItems first if they are not lists themselves (that gives problems) // as you will see, I also removed things like \n because that gave errors and added an empty value for parent_id if there is none in the data (otherwise it would mix up the data)
The code to import the .mongo file into R and then parse it into CSV:
library(rjson)
###### set working directory to write out the data file
setwd("/your/favourite/dir/json to csv/")
#never ever convert strings to factors
options(stringsAsFactors = FALSE)
#import the .mongo file to R
temp.data = fromJSON(file="temp.mongo", method="C", unexpected.escape="error")
file.remove("temp.csv") ## removes the old datafile if there is one
## (so the data is not appended to the file,
## but a new file is created)
listItem = temp.data[[1]] ## prepare the listItem the first time
for (listItem in temp.data){
parent_id = ""
if (length(listItem$parent_id)>0){
parent_id = listItem$parent_id
}
write.table(t(c(
listItem$votes$up_count, listItem$visible, parent_id,
gsub("\n", "", listItem$body), listItem$course_id, unlist(listItem["_type"]),
listItem$endorsed, listItem$anonymous, listItem$author_id,
unlist(listItem$comment_thread_id), listItem$author_username,
as.POSIXct(unlist(listItem$created_at)/1000, origin="1970-01-01"))), # end t(), c()
file="temp.csv", sep="\t", append=TRUE, row.names=FALSE, col.names=FALSE)
}