I want to extract JSON block where it satisfies multiple conditions. For example, extract a block which has variables with two or more desired value. Please see below given example.
[
{
id:"1",
name:"ABC - Project 1",
appName:"XYZ",
state:"New",
appType:"owner",
date:"May 12"
},
{
id:"2",
name:"DEF - Project 2",
appName:"UVW",
state:"In Progress",
appType:"manager",
date:"May 13"
},
{
id:"3",
name:"GHI - Project 3",
appName:"RST",
state:"In Progress",
appType:"owner",
date:"May 12"
},
{
id:"4",
name:"JKL - Project 4",
appName:"OPQ",
state:"Expired",
appType:"entity owner",
date:"July 13"
}
]
From the above JSON, I want to extract the JSON block where state:"In Progress" or state:"New", either of these states and it should of appType:"Owner" along with name:... Project 1; i.e. the following blocks as output:
{
id:"1",
name:"ABC - Project 1",
appName:"XYZ",
state:"New",
appType:"owner",
date:"May 12"
}
Which JSON Path expressions whould I use to extract:
$.[?((#.state == "In Progress" || #.state == "New") && #.appType== "owner" && <some regex expression>)]
But it doesn't extract any result. Is there any way to use multiple conditions for "OR/||" and "AND/&&" condition to extract that particular block. Please help!
Thanks,
Sid
How about:
$.[?((#.state == "In Progress" || #.state == "New") && #.appType== "owner" && #.name =~ /.*Project 1.*/)]
=~ is a Filter Operator allowing specifying a regular expression as the filter criteria
Demo:
More information: JMeter's JSON Path Extractor Plugin - Advanced Usage Scenarios
I'm using SublimeText2.
How to reindent Json code with a shortcut?
I've already installed packageControl and it works.
I already tried JsonReindent package but it has not a shortcut by default and i don't know its command name.
I already have this command (from my previous sublimetext2 on another pc, where it works fine):
// JSON reindent
{ "keys": ["ctrl+shift+j", "ctrl+shift+j"], "command": "json_reindent" },
But here it doesn't work.
I have come across this issue too. I tried to figure it out by install jsFormat as follows:
Open Package Control panel command+shift+p (mac), install package.
Install jsFormat.
Edit the shortcut for the jsformat:
Choose Key-Binding user, then edit
[
{
"keys": [
"command+l"
],
"command": "reindent"
},
{
"keys": [
"ctrl+shift+s"
],
"command": "auto_save"
},
{
"keys": [
"ctrl+alt+f"
],
"command": "js_format"
}
]
So next time when you try to format json code, just use ctrl+option+f for mac. Not mac, just use ctrl+alt+f
That's all.
Install this packages via CMD + SHIFT + P > Install package > Search for Pretty JSON and install.
And then turn ugly json via CMD + CTRL + J (OSX) CTRL + ALT + J (Windows/Linux) in pretty JSON!
Source is on GitHub.
Credit
For sublime 4 below work for me on MAC OS :
Install "Pretty JSON" sublime 4 by refer below:
Use key combination Cmd + Shift + p and type Install Package
Select Package Control: Install Package from the search results
Type Pretty JSON and select it from the search results
OR use "Manual Installation" steps of below link
https://packagecontrol.io/packages/Pretty%20JSON
now for shortcut :
Click on Sublime -> Preferences -> Key Bindings
Here add the below json part inside existing json and save the file
{ "keys": [ "ctrl+shift+j" ], "command": "pretty_json" }
you can use cmd too instead of ctrl, set says as per your preference
Complete json will look like below:
[{
"keys": [
"command+l"
],
"command": "reindent"
},
{
"keys": [
"ctrl+shift+s"
],
"command": "auto_save"
},
{
"keys": [
"ctrl+shift+k"
],
"command": "js_format"
},
{
"keys": [
"ctrl+shift+j"
],
"command": "pretty_json"
}
]
Now use below short key anytime to beautify json on sublime
ctrl+shift+j
You can also use command instead of ctrl
For Mac OS follow the steps:
Install Pretty JSON
Use key combination Cmd + Shift + p and type Install Package
Select Package Control: Install Package from the search results
Type Pretty JSON and select it from the search results
Add Key Binding
Open key bindings from Sublime Text > Preferences > Key Bindings
Add following key short cut in key bindings
{ "keys": [ "command+shift+j" ], "command": "pretty_json" }
Use Pretty JSON
Select JSON text and use key combination Cmd + Shift + j
If you look at the source, the name of the sublime_plugin.TextCommand subclass is SublimeJsonReindentCommand. Per the conventions for command names, the resulting command is named sublime_json_reindent. Simply change your key binding to:
{ "keys": ["ctrl+shift+j", "ctrl+shift+j"], "command": "sublime_json_reindent" },
and you should be all set.
You can use Command Palette to format a text
//Call Command Palette
Ctrl+Shift+P
//Format code instead of CMD + CTRL + J (OSX)
pretty format
I am using Linux OS. I found the solution first installing the package controller then Pretty JSON package.
Install the package controller for sublime here
Install Pretty JSON packages via CMD + SHIFT + P > Install package > Search for Pretty JSON and install. Details are here.
Open the Command Palette: Press Ctrl+Shift+P (Windows) or Cmd+Shift+P (OS X). Type 'install' in the Command Palette input box, which should autocomplete to 'Install Package Control'.
Then go to Sublime Text -> Preferences -> Package Control -> Type Package Control: Install Package in the pop up box which would open a separate pop up box and in that type the package you want to install in this case "Pretty JSON"
Turn ugly json to pretty JSON via CMD + CTRL + J (OS X)
Since version 4 it is available under the Selection -> Format -> JSON menu option.
I have a preference for shell commands to get things done. I have a very, very big file -- about 2.8 GB and the content is that of JSON. Everything is on one line, and I was told there are at least 1.5 million records in there.
I must prepare the file for consumption. Each record must be on its own line. Sample:
{"RomanCharacters":{"Alphabet":[{"RecordId":"1",...]},{"RecordId":"2",...},{"RecordId":"3",...},{"RecordId":"4",...},{"RecordId":"5",...} }}
Or, use the following...
{"Accounts":{"Customer":[{"AccountHolderId":"9c585258-c94c-442b-a2f0-1ebbcc274795","Title":"Mrs","Forename":"Tina","Surname":"Wright","DateofBirth":"1988-01-01","Contact":[{"Contact_Info":"9168777943","TypeId":"Mobile Number","PrimaryFlag":"No","Index":"1","Superseded":"No" },{"Contact_Info":"9503588153","TypeId":"Home Telephone","PrimaryFlag":"Yes","Index":"2","Superseded":"Yes" },{"Contact_Info":"acne.pimple#microchimerism.com","TypeId":"Email Address","PrimaryFlag":"No","Index":"3","Superseded":"No" },{"Contact_Info":"swati.singh#microchimerism.com","TypeId":"Email Address","PrimaryFlag":"Yes","Index":"4","Superseded":"Yes" }, {"Contact_Info":"christian.bale#hollywood.com","TypeId":"Email Address","PrimaryFlag":"No","Index":"5","Superseded":"NO" },{"Contact_Info":"15482475584","TypeId":"Mobile_Phone","PrimaryFlag":"No","Index":"6","Superseded":"No" }],"Address":[{"AddressPtr":"5","Line1":"Flat No.14","Line2":"Surya Estate","Line3":"Baner","Line4":"Pune ","Line5":"new","Addres_City":"pune","Country":"India","PostCode":"AB100KP","PrimaryFlag":"No","Superseded":"No"},{"AddressPtr":"6","Line1":"A-602","Line2":"Viva Vadegiri","Line3":"Virar","Line4":"new","Line5":"banglow","Addres_City":"Mumbai","Country":"India","PostCode":"AB10V6T","PrimaryFlag":"Yes","Superseded":"Yes"}],"Account":[{"Field_A":"6884133655531279","Field_B":"887.07","Field_C":"A Loan Product",...,"FieldY_":"2015-09-18","Field_Z":"24275627"}]},{"AccountHolderId":"92a5788f-cd8f-423d-ae5f-4eb0ceb457fd","_Title":"Dr","_Forename":"Christopher","_Surname":"Carroll","_DateofBirth":"1977-02-02","Contact":[{"Contact_Info":"9168777943","TypeId":"Mobile Number","PrimaryFlag":"No","Index":"7","Superseded":"No" },{"Contact_Info":"9503588153","TypeId":"Home Telephone","PrimaryFlag":"Yes","Index":"8","Superseded":"Yes" },{"Contact_Info":"acne.pimple#microchimerism.com","TypeId":"Email Address","PrimaryFlag":"No","Index":"9","Superseded":"No" },{"Contact_Info":"swati.singh#microchimerism.com","TypeId":"Email Address","PrimaryFlag":"Yes","Index":"10","Superseded":"Yes" }],"Address":[{"AddressPtr":"11","Line1":"Flat No.14","Line2":"Surya Estate","Line3":"Baner","Line4":"Pune ","Line5":"new","Addres_City":"pune","Country":"India","PostCode":"AB11TXF","PrimaryFlag":"No","Superseded":"No"},{"AddressPtr":"12","Line1":"A-602","Line2":"Viva Vadegiri","Line3":"Virar","Line4":"new","Line5":"banglow","Addres_City":"Mumbai","Country":"India","PostCode":"AB11O8W","PrimaryFlag":"Yes","Superseded":"Yes"}],"Account":[{"Field_A":"4121879819185553","Field_B":"887.07","Field_C":"A Loan Product",...,"Field_X":"2015-09-18","Field_Z":"25679434"}]},{"AccountHolderId":"4aa10284-d9aa-4dc0-9652-70f01d22b19e","_Title":"Dr","_Forename":"Cheryl","_Surname":"Ortiz","_DateofBirth":"1977-03-03","Contact":[{"Contact_Info":"9168777943","TypeId":"Mobile Number","PrimaryFlag":"No","Index":"13","Superseded":"No" },{"Contact_Info":"9503588153","TypeId":"Home Telephone","PrimaryFlag":"Yes","Index":"14","Superseded":"Yes" },{"Contact_Info":"acne.pimple#microchimerism.com","TypeId":"Email Address","PrimaryFlag":"No","Index":"15","Superseded":"No" },{"Contact_Info":"swati.singh#microchimerism.com","TypeId":"Email Address","PrimaryFlag":"Yes","Index":"16","Superseded":"Yes" }],"Address":[{"AddressPtr":"17","Line1":"Flat No.14","Line2":"Surya Estate","Line3":"Baner","Line4":"Pune ","Line5":"new","Addres_City":"pune","Country":"India","PostCode":"AB12SQR","PrimaryFlag":"No","Superseded":"No"},{"AddressPtr":"18","Line1":"A-602","Line2":"Viva Vadegiri","Line3":"Virar","Line4":"new","Line5":"banglow","Addres_City":"Mumbai","Country":"India","PostCode":"AB12BAQ","PrimaryFlag":"Yes","Superseded":"Yes"}],"Account":[{"Field_A":"3288214945919484","Field_B":"887.07","Field_C":"A Loan Product",...,"Field_Y":"2015-09-18","Field_Z":"66264768"}]}]}}
Final outcome should be:
{"RomanCharacters":{"Alphabet":[{"RecordId":"1",...]},
{"RecordId":"2",...},
{"RecordId":"3",...},
{"RecordId":"4",...},
{"RecordId":"5",...} }}
Attempted commands:
sed -e 's/,{"RecordId"/}]},\n{"RecordId"/g' sample.dat
awk '{gsub(",{\"RecordId\"",",\n{\"RecordId\"",$0); print $0}' sample.dat
The attempted commands works perfectly fine for small files. But it does not work for the 2.8 GB file that I must manipulate. Sed quits midway after 10 mins without reason and nothing was done. Awk errored with a Segmentation Fault (core dump) reason after many hours in. I tried perl's search and replace and got an error saying "Out of memory".
Any help/ ideas would be great!
Additional info on my machine:
More than 105 GB disk space available.
8 GB memory
4 cores CPU
Running Ubuntu 14.04
Since you've tagged your question with sed, awk AND perl, I gather that what you really need is a recommendation for a tool. While that's kind of off-topic, I believe that jq is something you could use for this. It will be better than sed or awk because it actually understands JSON. Everything shown here with jq could also be done in perl with a bit of programming.
Assuming content like the following (based on your sample):
{"RomanCharacters":{"Alphabet": [ {"RecordId":"1","data":"data"},{"RecordId":"2","data":"data"},{"RecordId":"3","data":"data"},{"RecordId":"4","data":"data"},{"RecordId":"5","data":"data"} ] }}
You can easily reformat this to "prettify" it:
$ jq '.' < data.json
{
"RomanCharacters": {
"Alphabet": [
{
"RecordId": "1",
"data": "data"
},
{
"RecordId": "2",
"data": "data"
},
{
"RecordId": "3",
"data": "data"
},
{
"RecordId": "4",
"data": "data"
},
{
"RecordId": "5",
"data": "data"
}
]
}
}
And we can dig in to the data to retrieve only the records you're interested in (regardless of what they're wrapped in):
$ jq '.[][][]' < data.json
{
"RecordId": "1",
"data": "data"
}
{
"RecordId": "2",
"data": "data"
}
{
"RecordId": "3",
"data": "data"
}
{
"RecordId": "4",
"data": "data"
}
{
"RecordId": "5",
"data": "data"
}
This is much more readable, both by humans and by tools like awk which process content line-by-line. If you want to join your lines for processing per your question, the awk becomes much more simple:
$ jq '.[][][]' < data.json | awk '{printf("%s ",$0)} /}/{printf("\n")}'
{ "RecordId": "1", "data": "data" }
{ "RecordId": "2", "data": "data" }
{ "RecordId": "3", "data": "data" }
{ "RecordId": "4", "data": "data" }
{ "RecordId": "5", "data": "data" }
Or, as #peak suggested in comments, eliminate the awk portion of thie entirely by using jq's -c (compact output) option:
$ jq -c '.[][][]' < data.json
{"RecordId":"1","data":"data"}
{"RecordId":"2","data":"data"}
{"RecordId":"3","data":"data"}
{"RecordId":"4","data":"data"}
{"RecordId":"5","data":"data"}
Regarding perl: Try setting the input line separator $/ to }, like this:
#!/usr/bin/perl
$/= "},";
while (<>){
print "$_\n";
}'
or, as a one-liner:
$ perl -e '$/="},";while(<>){print "$_\n"}' sample.dat
Try using } as the record separator, e.g. in Perl:
perl -l -0175 -ne 'print $_, $/' < input
You might need to glue back lines containing only }.
This avoids the memory problem by not looking at the data as a single record, but may go too far the other way with respect to performance (processing a single character at a time). Also note that it requires gawk for the built-in RT variable (value of the current record separator):
$ cat j.awk
BEGIN { RS="[[:print:]]" }
RT == "{" { bal++}
RT == "}" { bal-- }
{ printf "%s", RT }
RT == "," && bal == 2 { print "" }
END { print "" }
$ gawk -f j.awk j.txt
{"RomanCharacters":{"Alphabet":[{"RecordId":"1",...]},
{"RecordId":"2",...},
{"RecordId":"3",...},
{"RecordId":"4",...},
{"RecordId":"5",...} }}
Using the sample data provided here (the one that begins with {Accounts:{Customer... ), the solution to this problem is one that reads in the file and as it is reading it is counting the number of delimiters defined in $/. For every count of 10,000 delimiters, it will write out to a new file. And for each delimiter found, it gives it a new line. Here is how the script looks like:
#!/usr/bin/perl
$base="/home/dat789/incoming";
#$_="sample.dat";
$/= "}]},"; # delimiter to find and insert new line after
$n = 0;
$match="";
$filecount=0;
$recsPerFile=10000; # set number of records in a file
print "Processing " . $_ ."\n";
while (<>){
if ($n < $recsPerFile) {
$match=$match.$_."\n";
$n++;
print "."; #This is so that we'd know it has done something
}
else {
my $newfile="partfile".$recsPerFile."-".$filecount . ".dat";
open ( OUTPUT,'>', $newfile );
print OUTPUT $match;
$match="";
$filecount++;
$n=0;
print "Wrote file " . $newfile . "\n";
}
}
print "Finished\n\n";
I've used this script against the big 2.8 GB file where it's content is an unformatted one-liner JSON. The resulting output files would be missing the correct JSON headers and footers but this can be easily fixed.
Thank you so much guys for contributing!
I have a sublime text build system for running latex. I am using a shell script (with no arguments) to delete the various extensions that are generated by latex.
The problem is that sublime-text thinks filename.tex is the argument to the script.
So I tried the following script:
{
// General settings
"target": "make_pdf",
"selector": "text.tex.latex",
"cmd": ["latexmk", "-e","\\$dvipdf = 'dvipdfmx %O -o %D %S'", "-e", "\\$latex = 'latex %O -interaction=nonstopmode -synctex=1 %S'","-f", "-pdfdvi"],
"variants":
[
{ "cmd":["my_script.sh"],
"name": "clean"
}
]
}
Here I used ls in place of the script to illustrate. When I run the build-file, it tries to run
my_script.sh filename.tex
instead of just my_script.sh. What can I do to run just that from the build file ?
I have been there and done that . . . there and back again . . . and visited again and again . . .
The solution is to use a custom plugin (not necessarily relying upon a .sublime-build, but that is possible too) so that you can refer to the *.tex file that is open. I have some solutions you are probably unaware of, one of which is my own, and one where I modify the popular plugin LaTexTools.
latexmk cleans with a big -C (everything) or a little -c (some things that are pre-defined, and additional things that can be user-defined) -- so there is no need to use a separate custom cleaner script.
https://github.com/lawlist/LaTexTools-Unofficial-Modification
https://github.com/lawlist/ST2-plugin-latexmk-save-build-clean
FYI: I recommend putting latex (or something like that) in your subject line of the question -- I almost missed your question . . . and just happened to see the short summary and realized I knew something about this issue.
SAMPLE -- for MultiTaskBuild plugin located here: https://github.com/bizoo/MultiTaskBuild
{
"cmd": {
"latexmk -pvc . . .": {
"cmd": ["latexmk", "-r", "/Users/HOME/.latexmkrc", "$file"]
},
"latexmk -pv . . .": {
"cmd": ["latexmk",
"-e", "\\$pdflatex = 'pdflatex -enable-write18 %O -interaction=nonstopmode -synctex=1 %S'",
"-recorder-", "-pvc-", "-f", "-pdf", "-pv", "$file"]
},
"latexmk -c": {
"cmd": ["latexmk", "-c", "$file"]
},
"latexmk -C": {
"cmd": ["latexmk", "-C", "$file"]
}
},
"path": "$PATH:/usr/texbin:/usr/local/bin",
"file_regex": "^(...*?):([0-9]+): ([0-9]*)([^\\.]+)",
"selector": "text.tex.latex",
"default_task": "latexmk -pv . . .",
"target": "multi_task_exec"
}