Perl add content to JSON data after processing - json

I have a json file, which I am processing using perl JSON module.
Once I process it, I want to insert some content to in it.
Here is my input json file:
{
"sequence" : [
{
"type" : "event",
"attribute" : {
"contentText" : "Test Content",
"contentNumber" : "11"
}
}
],
"current" : 0,
"next" : 1
}
And below is my script:
#!/usr/bin/perl
use strict;
use warnings;
use JSON;
use Data::Dumper;
my $needed = 2;
my $filename = "test_file.json";
my $json_text = do {
open(my $json_fh, "<:encoding(UTF-8)", $filename)
or die("Can't open \$filename\": $!\n");
local $/;
<$json_fh>
};
my $json = JSON->new;
my $data = $json->decode($json_text);
my $aref = $data->{sequence};
print Dumper($aref);
my $number;
for my $element (#$aref) {
$number = $element->{attribute}->{contentNumber}."\n";
}
print "Number:$number\n";
my $total = $number + $needed;
foreach my $each_number ($number+1..$total){
print $each_number."\n";
}
print Dumper $data;
So what I needed over here is fetch contentNumber from given json file and increment value by 1 till $needed is mentioned and form a new json file.
And finally it should form JSON file which should have content like below:
Where whatever $needed variable value is mentioned that many times the json should form the data including the initial data.
{
"sequence" : [
{
"type" : "event",
"attribute" : {
"contentText" : "Test Content",
"contentNumber" : "11"
}
},
{
"type" : "event",
"attribute" : {
"contentText" : "Test Content",
"contentNumber" : "12"
}
},
{
"type" : "event",
"attribute" : {
"contentText" : "Test Content",
"contentNumber" : "13"
}
}
],
"current" : 0,
"next" : 1
}
I was thinking to push the data in foreach loop. But no clue how we can put it in data object which should give me an output with json format.

From the desired output it appears that you need the hashref which is in sequence's array. Then you need to add $needed number of its copies to that array, with contentNumber incremented in each. (I can't reconcile that with the shown code and I'll go with the desired output, which seems clear.)
Don't forget that the copies must be deep copies;† here I use dclone from Storable for that.
use Storable qw(dclone);
...
my $seq_href = dclone( $data->{sequence}[0] );
for (1..$needed) {
++$seq_href->{attribute}{contentNumber};
push #{$data->{sequence}}, dclone( $seq_href );
}
my $new_json_string = $json->encode($data); # then write it to file
This produces the desired output JSON in my tests.
† A variable or data structure containing references cannot be copied into a new, independent one by merely assigning
my #copy = #ary; # oups ... any references in there?
The problem is that when the elements that are references in #ary are copied into #copy then those elements in #copy, being the same references, point to same memory locations as the ones from #ary! So #copy and #ary are by no means independent -- they share data.
Sometimes that may be desired but if we need an independent copy, like in this problem, then we need to follow those references all the way and actually copy the data so that the copied structure indeed has its own data. And there are modules that do that of course.
Complex (nested) data structures by definition have references for elements and so we certainly can't get independent copies by one top-level assignment.
This is a very skinny description of a potentially sneaky and subtle bug. I'd suggest to read up more on it. One resource that comes up is an Effective Perler article.

Related

How to convert key value text to json arrya format python

I have a use case where we have text file like key value format .
The file is not any of the fixed format but created like key value .
We need to create JSON out of that file .
I am able to create JSON but when text format has array like structure it creates just Key value json not the array json structure .
This is my Input .
[DOCUMENT]
Headline=This is Headline
MainLanguage=EN
DocType.MxpCode=1000
Subject[0].MxpCode=BUSNES
Subject[1].MxpCode=CONS
Subject[2].MxpCode=ECOF
Author[0].MxpCode=6VL6
Industry[0].CtbCode=53
Industry[1].CtbCode=5340
Industry[2].CtbCode=534030
Industry[3].CtbCode=53403050
Symbol[0].Name=EXPE.OQ
Symbol[1].Name=ABNB.OQ
WorldReg[0].CtbCode=G4
Country[0].CtbCode=G26
Country[1].CtbCode=G2V
[ENDOFFILE]
Exiting code to create json is below
with open("file1.csv") as f:
lines = f.readlines()
data = {}
for line in lines:
parts = line.split('=')
if len(parts) == 2:
data[parts[0].strip()] = parts[1].strip()
print(json.dumps(data, indent=' '))
The current output is below
{
"Headline": "This is Headline",
"MainLanguage": "EN",
"DocType.MxpCode": "1000",
"Subject[0].MxpCode": "BUSNES",
"Subject[1].MxpCode": "CONS",
"Subject[2].MxpCode": "ECOF",
"Author[0].MxpCode": "6VL6",
"Industry[0].CtbCode": "53",
"Industry[1].CtbCode": "5340",
"Industry[2].CtbCode": "534030",
"Industry[3].CtbCode": "53403050",
"Symbol[0].Name": "EXPE.OQ",
"Symbol[1].Name": "ABNB.OQ",
"WorldReg[0].CtbCode": "G4",
"Country[0].CtbCode": "G26",
"Country[1].CtbCode": "G2V"
}
Expected out is is something like below
For the Subject key and like wise for others also
{
"subject": [
{
"mxcode": 123
},
{
"mxcode": 123
},
{
"mxcode": 123
}
]
}
Like wise for Industry and Symbol and Country.
so the idea is when we have position in the text file it should be treated as array in the json output .
Use one more loop as it is nested. Use for loop from where subject starts. try it that way.

How do I recursively search a JSON file for all nodes matching a given pattern and return the JSON 'path' to the node and it's value?

Say I have this JSON in a text file:
{"widget": {
"debug": "on",
"window": {
"title": "Sample Konfabulator Widget",
"name": "main_window",
"width": 500,
"height": 500
},
"image": {
"src": "Images/Sun.png",
"name": "sun1",
"hOffset": 250,
"vOffset": 250,
"alignment": "center"
},
"text": {
"data": "Click Here",
"size": 36,
"style": "bold",
"name": "text1",
"hOffset": 250,
"vOffset": 100,
"alignment": "center",
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;"
}
}}
Using Perl I have read the file into a JSON object called $json_obj using JSON::XS.
How do I search $json_obj for all nodes called name and return/print the following as the result/output:
widget->window->name: main_window
widget->image->name: sun1
widget->text->name: text1
Notes:
node names matching the search term could appear at any level of the tree
search terms could be plain text or a regular expression
I'd like to be able to supply my own branch separator to override a default of, say, ->
example / (for simplicity, I'll just put this in a perl $variable)
I would like to be able to specify multiple node levels in my search, so as the specify a path to match, for example: specifying id/colour would return all paths that contain a node called id that is also a parent with a child node called colour
displaying double quotes around the result values is optional
I want to be able to search for multiple patterns, e.g. /(name|alignment)/ for "find all nodes called name or alignment
Example showing results of search in last note above:
widget->window->name: main_window
widget->image->name: sun1
widget->image->alignment: center
widget->text->name: text1
widget->text->alignment: center
Since JSON is mostly just text, I'm not yet sure of the benefit of even using JSON::XS so any advice on why this is better or worse is most welcome.
It goes without saying that it needs to be recursive so it can search n arbitrary levels deep.
This is what I have so far, but I'm only part way there:
#!/usr/bin/perl
use 5.14.0;
use warnings;
use strict;
use IO::File;
use JSON::XS;
my $jsonfile = '/home/usr/filename.json';
my $jsonpath = 'image/src'; # example search path
my $pathsep = '/'; # for displaying results
my $fh = IO::File->new("$jsonfile", "r");
my $jsontext = join('',$fh->getlines());
$fh->close();
my $jsonobj = JSON::XS->new->utf8->pretty;
if (defined $jsonpath) {
my $perltext = $jsonobj->decode($jsontext); # is this correct?
recurse_tree($perltext);
} else {
# print file to STDOUT
say $jsontext;
}
sub recurse_tree {
my $hash = shift #_;
foreach my $key (sort keys %{$hash}) {
if ($key eq $jsonpath) {
say "$key = %{$hash}{$key} \n"; # example output
}
if (ref $hash->{$key} eq 'HASH' ||
ref $hash->{$key} eq 'ARRAY') {
recurse_tree($hash->{$key});
}
}
}
exit;
The expected result from the above script would be:
widget/image/src: Images/Sun.png
Once that JSON is decoded, there is a complex (nested) Perl data structure that you want to search through, and the code you show is honestly aiming for that.
However, there are libraries out there which can help; either to do the job fully or to provide complete, working, and tested code that you can fine tune to the exact needs.
The module Data::Leaf::Walker seems suitable. A simple example
use warnings;
use strict;
use feature 'say';
use Data::Dump qw(dd);
use JSON;
use List::Util qw(any);
use Data::Leaf::Walker;
my $file = shift // 'data.json'; # provided data sample
my $json_data = do { local (#ARGV, $/) = $file; <> }; # read into a string
chomp $json_data;
my $ds = decode_json $json_data;
dd $ds; say ''; # show decoded data
my $walker = Data::Leaf::Walker->new($ds);
my $sep = '->';
while ( my ($key_path, $value) = $walker->each ) {
my #keys_in_path = #$key_path;
if (any { $_ eq 'name' } #keys_in_path) { # selection criteria
say join($sep, #keys_in_path), " => $value"
}
}
This 'walker' goes through the data structure, keeping the list of keys to each leaf. This is what makes this module particularly suitable for your quest, along with its simplicity of purpose in comparison to many others. See documentation.
The above prints, for the sample data provided in the question
widget->window->name => main_window
widget->text->name => text1
widget->image->name => sun1
The implementation of the criterion for which key-paths get selected in the code above is rather simple-minded, since it checks for 'name' anywhere in the path, once, and then prints the whole path. While the question doesn't specify what to do about matches earlier in the path, or with multiple ones, this can be adjusted since we always have the full path.
The rest of your wish list is fairly straightforward to implement as well. Peruse List::Util and List::MoreUtils for help with array analysis.
Another module, that is a great starting point for possible specific needs, is Data::Traverse. It is particularly simple, at 70-odd lines of code, so very easy to customize.
Depending on your task, you might consider using jq. This output is simple, but you can get as complex as you like:
$ jq -r '.. | .image? | .src | strings' test.json
Images/Sun.png
$ jq -r '.. | .name? | strings' test.json
main_window
sun1
text1
Walking the data structure isn't that bad, although it's a bit weird the first couple of times you do it. There are various modules on CPAN that will do for you (as zdim shows), but this is something you should probably know how to do on your own. We have some big examples in Intermediate Perl.
One way to do it is to start with a queue of things to process. This is iteration, not recursion, and depending on how you add elements to the queue, you can do either depth-first or breadth-first searches.
For each item, I'll track the path of keys to get there, and the sub-hash. That's the problem with your recursive approach: you don't allow for a way to track the path.
At the start, the queue has one item because we are at the top. I'll also define a target key, since your problem has that:
my #queue = ( { key_path => [], node => $hash } );
my $target = 'name';
Next, I process every item in the queue (the while). I expect each value of node to be a hash, so I'll get all the keys of that hash (the foreach). This represents the next level of the hash.
Inside the foreach, I make a new key path with the one that exists along with the one I'm processing. I also get the next value by using that key.
After that, I can do task specific processing. If I've found my target key, I'll do whatever I need to do. In this case I output some stuff, but I could add to a different data structure and so on. I use next to stop processing that key (although I could keep going). If I didn't find the target key, I make another entry in the queue if the value is another hash reference.
Then, I go back to processing the queue.
use v5.24; # use postfix dereferencing
while( my $h = shift #queue ) {
foreach my $next_key ( keys $h->{node}->%* ) {
my $key_path = [ $h->{key_path}->#*, $next_key ];
my $value = $h->{node}{$next_key};
if( $next_key eq $target ) {
say join( "->", $key_path->#* ), " = $value";
next;
}
elsif( ref $value eq ref {} ) {
push #queue, { key_path => $key_path, node => $value };
}
}
}
I end up with output like:
widget->text->name = text1
widget->image->name = sun1
widget->window->name = main_window
From there, you can customize this to get the other features you need. If you want to find a complex key, you just do a little more work to compare the key path to what you want.

What JSON format does STRIP_OUTER_ARRAY support?

I have a file composed of a single array containing multiple records.
{
"Client": [
{
"ClientNo": 1,
"ClientName": "Alpha",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "12345"
},
{
"BusinessNo": 2,
"IndustryCode": "23456"
}
]
},
{
"ClientNo": 2,
"ClientName": "Bravo",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "34567"
},
{
"BusinessNo": 2,
"IndustryCode": "45678"
}
]
}
]
}
I load it with the following code:
create or replace stage stage.test
url='azure://xxx/xxx'
credentials=(azure_sas_token='xxx');
create table if not exists stage.client (json_data variant not null);
copy into stage.client_test
from #stage.test/client_test.json
file_format = (type = 'JSON' strip_outer_array = true);
Snowflake imports the entire file as one row.
I would like the the COPY INTO command to remove the outer array structure and load the records into separate table rows.
When I load larger files, I hit the size limit for variant and get the error Error parsing JSON: document is too large, max size 16777216 bytes.
If you can import the file into Snowflake, into a single row, then you can use LATERAL FLATTEN on the Clients field to generate one row per element in the array.
Here's a blog post on LATERAL and FLATTEN (or you could look them up in the snowflake docs):
https://support.snowflake.net/s/article/How-To-Lateral-Join-Tutorial
If the format of the file is, as specified, a single object with a single property that contains an array with 500 MB worth of elements in it, then perhaps importing it will still work -- if that works, then LATERAL FLATTEN is exactly what you want. But that form is not particularly great for data processing. You might want to use some text processing script to massage the data if that's needed.
RECOMMENDATION #1:
The problem with your JSON is that it doesn't have an outer array. It has a single outer object containing a property with an inner array.
If you can fix the JSON, that would be the best solution, and then STRIP_OUTER_ARRAY will work as you expected.
You could also try to recompose the JSON (an ugly business) after reading line for line with:
CREATE OR REPLACE TABLE X (CLIENT VARCHAR);
COPY INTO X FROM (SELECT $1 CLIENT FROM #My_Stage/Client.json);
User Response to Recommendation #1:
Thank you. So from what I gather, COPY with STRIP_OUTER_ARRAY can handle a file starting and ending with square brackets, and parse the file as if they were not there.
The real files don't have line breaks, so I can't read the file line by line. I will see if the source system can change the export.
RECOMMENDATION #2:
Also if you would like to see what the JSON parser does, you can experiment using this code, I have parsed JSON on the copy command using similar code. Working with your JSON data in small project can help you shape the Copy command to work as intended.
CREATE OR REPLACE TABLE SAMPLE_JSON
(ID INTEGER,
DATA VARIANT
);
INSERT INTO SAMPLE_JSON(ID,DATA)
SELECT
1,parse_json('{
"Client": [
{
"ClientNo": 1,
"ClientName": "Alpha",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "12345"
},
{
"BusinessNo": 2,
"IndustryCode": "23456"
}
]
},
{
"ClientNo": 2,
"ClientName": "Bravo",
"ClientBusiness": [
{
"BusinessNo": 1,
"IndustryCode": "34567"
},
{
"BusinessNo": 2,
"IndustryCode": "45678"
}
]
}
]
}');
SELECT
C.value:ClientNo AS ClientNo
,C.value:ClientName::STRING AS ClientName
,ClientBusiness.value:BusinessNo::Integer AS BusinessNo
,ClientBusiness.value:IndustryCode::Integer AS IndustryCode
from SAMPLE_JSON f
,table(flatten( f.DATA,'Client' )) C
,table(flatten(c.value:ClientBusiness,'')) ClientBusiness;
User Response to Recommendation #2:
Thank you for the parse_json example!
Trouble is, the real files are sometimes 500 MB, so the parse_json function chokes.
Follow-up on Recommendation #2:
The JSON needs to be in the NDJSON http://ndjson.org/ format. Otherwise the JSON will be impossible to parse because of the potential for large files.
Hope the above helps other running into similar questions!

Perl Decode JSON - Not an ARRAY referencce

Getting error "Not an ARRAY reference" when trying to print my hash. Not sure where I am going wrong. IS there a way to print the decoded json in such a way that it makes identfying the hash elements easier?
use JSON qw( decode_json );
my $owm_json= <<'__EOI__';
{"cod":"200","city":{"id":"4270936","name":"Bazine","coord":{"lon":-99.6935,"lat":38.4421},"country":"United States of America","population":0},"cnt":65,"model":"OWM","list":{"0":{"dt":1390615200,"main":{"temp":39.63,"temp_min":39.69,"temp_max":43.7,"pressure":950.95,"humidity":62.4},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":73,"low":0,"middle":6,"high":72},"wind":{"speed":15.98,"deg":269,"gust":13.8},"sys":{"pod":"n"}},"1":{"dt":1390626000,"main":{"temp":40.89,"temp_min":39.69,"temp_max":43.7,"pressure":951.79,"humidity":54.7},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":82,"low":0,"middle":3,"high":82},"wind":{"speed":16.78,"deg":290,"gust":16.5},"sys":{"pod":"n"}},"2":{"dt":1390636800,"main":{"temp":41.45,"temp_min":40.91,"temp_max":41.72,"pressure":952.77,"humidity":47.7},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03n"}],"clouds":{"all":35,"low":0,"middle":0,"high":35},"wind":{"speed":16.63,"deg":314,"gust":18.7},"sys":{"pod":"n"}},"3":{"dt":1390647600,"main":{"temp":39.81,"temp_min":39.83,"temp_max":41.72,"pressure":953.7,"humidity":48.9},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03n"}],"clouds":{"all":32,"low":0,"middle":0,"high":32},"wind":{"speed":14.13,"deg":314,"gust":16.5},"sys":{"pod":"n"}},"4":{"dt":1390658400,"main":{"temp":40.51,"temp_min":38.5,"temp_max":40.53,"pressure":954.95,"humidity":50.3},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"clouds":{"all":23,"low":0,"middle":0,"high":23},"wind":{"speed":12.79,"deg":318,"gust":13.9},"sys":{"pod":"d"}},"5":{"dt":1390669200,"main":{"temp":51.67,"temp_min":38.5,"temp_max":51.66,"pressure":955.03,"humidity":44.3},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"clouds":{"all":18,"low":0,"middle":0,"high":18},"wind":{"speed":14.49,"deg":348,"gust":10.1},"sys":{"pod":"d"}},"6":{"dt":1390680000,"main":{"temp":54.99,"temp_min":51.75,"temp_max":55,"pressure":952.14,"humidity":40.3},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"clouds":{"all":1,"low":0,"middle":0,"high":1},"wind":{"speed":8.45,"deg":326,"gust":7.5},"sys":{"pod":"d"}},"7":{"dt":1390690800,"main":{"temp":46.15,"temp_min":45.82,"temp_max":55.35,"pressure":950.37,"humidity":53},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"02d"}],"clouds":{"all":10,"low":0,"middle":0,"high":10},"wind":{"speed":5.05,"deg":233,"gust":3.3},"sys":{"pod":"d"}},"8":{"dt":1390701600,"main":{"temp":45.1,"temp_min":45.1,"temp_max":47.71,"pressure":948.92,"humidity":53.8},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":12.77,"deg":235,"gust":9.8},"sys":{"pod":"n"}},"9":{"dt":1390712400,"main":{"temp":43.43,"temp_min":43.41,"temp_max":47.71,"pressure":947.31,"humidity":56.4},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":14.21,"deg":243,"gust":12.5},"sys":{"pod":"n"}},"10":{"dt":1390723200,"main":{"temp":41.38,"temp_min":41.43,"temp_max":43.41,"pressure":946.75,"humidity":59},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":14.52,"deg":257,"gust":13.9},"sys":{"pod":"n"}},"11":{"dt":1390734000,"main":{"temp":39.83,"temp_min":39.4,"temp_max":43.41,"pressure":946.52,"humidity":56.8},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":11.66,"deg":321,"gust":11.7},"sys":{"pod":"n"}},"12":{"dt":1390744800,"main":{"temp":41.22,"temp_min":39.16,"temp_max":41.25,"pressure":947.87,"humidity":50.3},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":9.37,"deg":327,"gust":8},"sys":{"pod":"d"}},"13":{"dt":1390755600,"main":{"temp":51.73,"temp_min":39.16,"temp_max":51.76,"pressure":947.62,"humidity":34.6},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":11.36,"deg":346,"gust":7.7},"sys":{"pod":"d"}},"14":{"dt":1390766400,"main":{"temp":53.58,"temp_min":51.85,"temp_max":53.64,"pressure":945.56,"humidity":30.5},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":12.48,"deg":345,"gust":8.4},"sys":{"pod":"d"}},"15":{"dt":1390777200,"main":{"temp":46.44,"temp_min":46.4,"temp_max":53.64,"pressure":948.09,"humidity":39},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01d"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":11.69,"deg":14,"gust":10.1},"sys":{"pod":"d"}},"16":{"dt":1390788000,"main":{"temp":37.4,"temp_min":37.36,"temp_max":46.4,"pressure":954.84,"humidity":54.9},"weather":[{"id":600,"main":"Snow","description":"light snow","icon":"13n"}],"clouds":{"all":66,"low":66,"middle":19,"high":0},"wind":{"speed":24.34,"deg":17,"gust":15.1},"snow":{"3h":0.12},"sys":{"pod":"n"}},"17":{"dt":1390798800,"main":{"temp":27.99,"temp_min":27.97,"temp_max":46.4,"pressure":959.47,"humidity":33.9},"weather":[{"id":600,"main":"Snow","description":"light snow","icon":"13n"}],"clouds":{"all":80,"low":63,"middle":57,"high":0},"wind":{"speed":21.56,"deg":10,"gust":12.7},"snow":{"3h":0.1},"sys":{"pod":"n"}},"18":{"dt":1390809600,"main":{"temp":22.53,"temp_min":22.55,"temp_max":27.95,"pressure":961.66,"humidity":27.8},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":99,"low":8,"middle":99,"high":0},"wind":{"speed":17.02,"deg":2,"gust":10.3},"sys":{"pod":"n"}},"19":{"dt":1390820400,"main":{"temp":20.61,"temp_min":20.57,"temp_max":27.95,"pressure":962.45,"humidity":23.1},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":100,"low":8,"middle":100,"high":0},"wind":{"speed":13.67,"deg":2,"gust":8.5},"sys":{"pod":"n"}},"20":{"dt":1390831200,"main":{"temp":20.32,"temp_min":20.01,"temp_max":20.55,"pressure":964.61,"humidity":21.3},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"clouds":{"all":100,"low":44,"middle":100,"high":0},"wind":{"speed":10.04,"deg":14,"gust":5.9},"sys":{"pod":"d"}},"21":{"dt":1390842000,"main":{"temp":23.4,"temp_min":20.01,"temp_max":23.4,"pressure":965.27,"humidity":17.9},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"clouds":{"all":100,"low":52,"middle":100,"high":0},"wind":{"speed":6.83,"deg":6,"gust":4.2},"sys":{"pod":"d"}},"22":{"dt":1390852800,"main":{"temp":25.48,"temp_min":23.4,"temp_max":25.54,"pressure":963.17,"humidity":17.4},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"clouds":{"all":100,"low":44,"middle":100,"high":0},"wind":{"speed":4.73,"deg":346,"gust":3.5},"sys":{"pod":"d"}},"23":{"dt":1390863600,"main":{"temp":24.8,"temp_min":23.4,"temp_max":25.63,"pressure":963.34,"humidity":19.6},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"clouds":{"all":100,"low":39,"middle":100,"high":0},"wind":{"speed":3.67,"deg":40,"gust":2},"sys":{"pod":"d"}},"24":{"dt":1390874400,"main":{"temp":23.34,"temp_min":23.36,"temp_max":24.73,"pressure":964.88,"humidity":21},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":100,"low":3,"middle":100,"high":0},"wind":{"speed":5.23,"deg":115,"gust":2.9},"sys":{"pod":"n"}},"25":{"dt":1390885200,"main":{"temp":20.53,"temp_min":20.53,"temp_max":24.73,"pressure":965.28,"humidity":25.9},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":96,"low":2,"middle":96,"high":0},"wind":{"speed":5.66,"deg":138,"gust":3.8},"sys":{"pod":"n"}},"26":{"dt":1390896000,"main":{"temp":18.52,"temp_min":18.54,"temp_max":20.08,"pressure":964.39,"humidity":41.1},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":1,"low":0,"middle":1,"high":0},"wind":{"speed":6.39,"deg":171,"gust":4.6},"sys":{"pod":"n"}},"27":{"dt":1390906800,"main":{"temp":18.68,"temp_min":18.41,"temp_max":20.08,"pressure":963.68,"humidity":67.5},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"02n"}],"clouds":{"all":4,"low":0,"middle":4,"high":0},"wind":{"speed":6.89,"deg":201,"gust":5},"sys":{"pod":"n"}},"28":{"dt":1390917600,"main":{"temp":23.43,"temp_min":18.88,"temp_max":23.41,"pressure":964.4,"humidity":75.5},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"clouds":{"all":37,"low":7,"middle":33,"high":0},"wind":{"speed":10.26,"deg":220,"gust":7.5},"sys":{"pod":"d"}},"29":{"dt":1390928400,"main":{"temp":33.31,"temp_min":18.88,"temp_max":33.31,"pressure":963.48,"humidity":41.3},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"clouds":{"all":45,"low":29,"middle":17,"high":0},"wind":{"speed":12.1,"deg":242,"gust":6.9},"sys":{"pod":"d"}},"30":{"dt":1390939200,"main":{"temp":37.06,"temp_min":33.48,"temp_max":37.02,"pressure":960.5,"humidity":31.6},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"clouds":{"all":12,"low":12,"middle":0,"high":0},"wind":{"speed":7.17,"deg":284,"gust":5},"sys":{"pod":"d"}},"31":{"dt":1390950000,"main":{"temp":27,"temp_min":26.96,"temp_max":37.02,"pressure":961.66,"humidity":53.9},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"02d"}],"clouds":{"all":6,"low":6,"middle":0,"high":0},"wind":{"speed":9.68,"deg":38,"gust":6.2},"sys":{"pod":"d"}},"32":{"dt":1390960800,"main":{"temp":17.92,"temp_min":18,"temp_max":26.92,"pressure":962.49,"humidity":78.9},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":9.59,"deg":66,"gust":6.8},"sys":{"pod":"n"}},"33":{"dt":1390971600,"main":{"temp":17.22,"temp_min":16.92,"temp_max":26.92,"pressure":961.43,"humidity":73.3},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":6.13,"deg":101,"gust":3.9},"sys":{"pod":"n"}},"34":{"dt":1390982400,"main":{"temp":20.05,"temp_min":17.28,"temp_max":19.99,"pressure":957.98,"humidity":60.7},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":7.78,"deg":161,"gust":5.9},"sys":{"pod":"n"}},"35":{"dt":1390993200,"main":{"temp":22.8,"temp_min":17.28,"temp_max":22.8,"pressure":954.89,"humidity":44.4},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"01n"}],"clouds":{"all":0,"low":0,"middle":0,"high":0},"wind":{"speed":11.58,"deg":196,"gust":10.5},"sys":{"pod":"n"}},"36":{"dt":1391004000,"main":{"temp":27.03,"temp_min":22.8,"temp_max":27.01,"pressure":952.39,"humidity":37.7},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"clouds":{"all":15,"low":0,"middle":0,"high":15},"wind":{"speed":16.48,"deg":206,"gust":15.1},"sys":{"pod":"d"}},"37":{"dt":1391014800,"main":{"temp":41.68,"temp_min":22.8,"temp_max":41.7,"pressure":950.31,"humidity":27.1},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"clouds":{"all":21,"low":0,"middle":0,"high":21},"wind":{"speed":19.14,"deg":217,"gust":12.7},"sys":{"pod":"d"}},"38":{"dt":1391025600,"main":{"temp":48.13,"temp_min":41.86,"temp_max":48.07,"pressure":946.47,"humidity":28.8},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"clouds":{"all":27,"low":0,"middle":0,"high":27},"wind":{"speed":15.62,"deg":221,"gust":9.6},"sys":{"pod":"d"}},"39":{"dt":1391036400,"main":{"temp":39.42,"temp_min":39.47,"temp_max":48.18,"pressure":945.8,"humidity":46.6},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"clouds":{"all":33,"low":0,"middle":0,"high":33},"wind":{"speed":9.76,"deg":217,"gust":7},"sys":{"pod":"d"}},"40":{"dt":1391047200,"main":{"temp":36.48,"temp_min":36.45,"temp_max":39.47,"pressure":944.9,"humidity":53.5},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03n"}],"clouds":{"all":44,"low":0,"middle":0,"high":44},"wind":{"speed":11.19,"deg":220,"gust":7.4},"sys":{"pod":"n"}},"41":{"dt":1391058000,"main":{"temp":35.64,"temp_min":35.65,"temp_max":39.47,"pressure":943.31,"humidity":52.6},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03n"}],"clouds":{"all":42,"low":0,"middle":0,"high":42},"wind":{"speed":9.31,"deg":230,"gust":6.4},"sys":{"pod":"n"}},"42":{"dt":1391068800,"main":{"temp":34.54,"temp_min":34.57,"temp_max":35.65,"pressure":942.67,"humidity":53.4},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":64,"low":0,"middle":0,"high":64},"wind":{"speed":9.03,"deg":252,"gust":6.1},"sys":{"pod":"n"}},"43":{"dt":1391079600,"main":{"temp":35.6,"temp_min":34.57,"temp_max":35.87,"pressure":943.94,"humidity":53.4},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":72,"low":0,"middle":0,"high":72},"wind":{"speed":9.91,"deg":328,"gust":6.7},"sys":{"pod":"n"}},"44":{"dt":1391090400,"main":{"temp":26.78,"temp_min":26.78,"temp_max":35.56,"pressure":947.55,"humidity":77.2},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"clouds":{"all":79,"low":0,"middle":0,"high":79},"wind":{"speed":18.3,"deg":24,"gust":12},"sys":{"pod":"d"}},"45":{"dt":1391101200,"main":{"temp":33.49,"temp_min":26.78,"temp_max":35.56,"pressure":949.78,"humidity":58},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"clouds":{"all":79,"low":0,"middle":18,"high":72},"wind":{"speed":8.21,"deg":45,"gust":5},"sys":{"pod":"d"}},"46":{"dt":1391112000,"main":{"temp":38.25,"temp_min":33.66,"temp_max":38.21,"pressure":948.38,"humidity":50.6},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"clouds":{"all":87,"low":0,"middle":2,"high":86},"wind":{"speed":6.8,"deg":85,"gust":3.5},"sys":{"pod":"d"}},"47":{"dt":1391122800,"main":{"temp":34.84,"temp_min":33.66,"temp_max":38.39,"pressure":948.29,"humidity":58.7},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"clouds":{"all":90,"low":0,"middle":1,"high":90},"wind":{"speed":5.96,"deg":99,"gust":3.6},"sys":{"pod":"d"}},"48":{"dt":1391133600,"main":{"temp":31.03,"temp_min":31.05,"temp_max":34.72,"pressure":948.27,"humidity":67.5},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":79,"low":0,"middle":0,"high":79},"wind":{"speed":8.53,"deg":126,"gust":6},"sys":{"pod":"n"}},"49":{"dt":1391144400,"main":{"temp":27.75,"temp_min":27.77,"temp_max":34.72,"pressure":946.79,"humidity":75.4},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":68,"low":0,"middle":0,"high":68},"wind":{"speed":8.86,"deg":135,"gust":6.6},"sys":{"pod":"n"}},"50":{"dt":1391155200,"main":{"temp":26.46,"temp_min":26.49,"temp_max":27.77,"pressure":944.93,"humidity":77.7},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":54,"low":0,"middle":1,"high":54},"wind":{"speed":7.32,"deg":126,"gust":5.3},"sys":{"pod":"n"}},"51":{"dt":1391166000,"main":{"temp":26.51,"temp_min":26.47,"temp_max":27.77,"pressure":943.57,"humidity":76.8},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04n"}],"clouds":{"all":71,"low":0,"middle":12,"high":71},"wind":{"speed":8.25,"deg":121,"gust":6.4},"sys":{"pod":"n"}},"52":{"dt":1391176800,"main":{"temp":28.17,"temp_min":26.35,"temp_max":28.13,"pressure":944.18,"humidity":72.3},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04d"}],"clouds":{"all":92,"low":1,"middle":57,"high":92},"wind":{"speed":5.34,"deg":98,"gust":3},"sys":{"pod":"d"}},"53":{"dt":1391187600,"main":{"temp":37.06,"temp_min":26.35,"temp_max":37.13,"pressure":943.56,"humidity":57.3},"weather":[{"id":803,"main":"Clouds","description":"broken clouds","icon":"04d"}],"clouds":{"all":62,"low":8,"middle":29,"high":57},"wind":{"speed":4.06,"deg":69,"gust":1.8},"sys":{"pod":"d"}},"54":{"dt":1391198400,"main":{"temp":43.34,"temp_min":37.15,"temp_max":43.36,"pressure":942.37,"humidity":52.1},"weather":[{"id":800,"main":"Clear","description":"sky is clear","icon":"02d"}],"clouds":{"all":7,"low":0,"middle":0,"high":6},"wind":{"speed":5.03,"deg":5,"gust":2.6},"sys":{"pod":"d"}},"55":{"dt":1391209200,"main":{"temp":38.35,"temp_min":37.15,"temp_max":43.99,"pressure":944.19,"humidity":60.7},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"clouds":{"all":38,"low":0,"middle":1,"high":38},"wind":{"speed":9.05,"deg":30,"gust":6.4},"sys":{"pod":"d"}},"56":{"dt":1391220000,"main":{"temp":33.87,"temp_min":33.89,"temp_max":38.34,"pressure":947.09,"humidity":66.8},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":90,"low":0,"middle":44,"high":90},"wind":{"speed":8.25,"deg":34,"gust":6.3},"sys":{"pod":"n"}},"57":{"dt":1391230800,"main":{"temp":32.59,"temp_min":32.14,"temp_max":38.34,"pressure":947.46,"humidity":66.6},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":90,"low":0,"middle":60,"high":89},"wind":{"speed":9.09,"deg":23,"gust":6.7},"sys":{"pod":"n"}},"58":{"dt":1391241600,"main":{"temp":30.47,"temp_min":30.43,"temp_max":32.81,"pressure":947.9,"humidity":69.8},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":96,"low":2,"middle":96,"high":52},"wind":{"speed":13.09,"deg":21,"gust":9.2},"sys":{"pod":"n"}},"59":{"dt":1391252400,"main":{"temp":27.23,"temp_min":27.09,"temp_max":32.81,"pressure":948.71,"humidity":81.7},"weather":[{"id":804,"main":"Clouds","description":"overcast clouds","icon":"04n"}],"clouds":{"all":85,"low":1,"middle":85,"high":31},"wind":{"speed":10.63,"deg":35,"gust":9.4},"sys":{"pod":"n"}},"60":{"dt":1391263200,"main":{"temp":26.02,"temp_min":26.06,"temp_max":27.59,"pressure":951.34,"humidity":93.6},"weather":[{"id":601,"main":"Snow","description":"snow","icon":"13d"}],"clouds":{"all":99,"low":67,"middle":99,"high":80},"wind":{"speed":9.85,"deg":37,"gust":6.8},"snow":{"3h":3.7},"sys":{"pod":"d"}},"61":{"dt":1391274000,"main":{"temp":27.5,"temp_min":26.04,"temp_max":27.54,"pressure":952.46,"humidity":87.2},"weather":[{"id":601,"main":"Snow","description":"snow","icon":"13d"}],"clouds":{"all":100,"low":82,"middle":99,"high":89},"wind":{"speed":7.8,"deg":42,"gust":4.6},"snow":{"3h":6.1},"sys":{"pod":"d"}},"62":{"dt":1391284800,"main":{"temp":29.44,"temp_min":27.77,"temp_max":29.46,"pressure":951.54,"humidity":82.5},"weather":[{"id":600,"main":"Snow","description":"light snow","icon":"13d"}],"clouds":{"all":100,"low":99,"middle":31,"high":49},"wind":{"speed":5.94,"deg":36,"gust":3.2},"snow":{"3h":0.1},"sys":{"pod":"d"}},"63":{"dt":1391295600,"main":{"temp":22.62,"temp_min":22.62,"temp_max":29.64,"pressure":952.76,"humidity":92.3},"weather":[{"id":601,"main":"Snow","description":"snow","icon":"13d"}],"clouds":{"all":89,"low":88,"middle":16,"high":25},"wind":{"speed":2.29,"deg":93,"gust":1.5},"snow":{"3h":6.2},"sys":{"pod":"d"}},"model":"gfs"}}";
__EOI__
my $decoded = decode_json($owm_json);
print $decoded->{list}[0]->{weather}[0]->{description};
Your string $owm_json isn't a valid Perl string. From the short segment:
"{"cod":"200","city":{"id":"4270936","name":
This should actually be:
"{\"cod\":\"200\",\"city\":{\"id\":\"4270936\",\"name\":
You need to escape the inner quotation marks. Try putting use strict at the beginning of the script and that should have caught this error.
Update from comments:
Looking at your JSON data, you do have an incorrect access to the descpiption field. Here's the section you are trying to display in your example:
"list" : {
"0" : {
"weather" : [{
"id" : 803,
"main" : "Clouds",
"description" : "broken clouds",
"icon" : "04n"
}
],
Notice that after "list" you have "0", not an array reference which would use the [] characters. In your script, after {list} use {0} and not [0] and that should fix your issue:
print $decoded->{list}{0}{weather}[0]{description};
The value of list is a hash (...,"list":{"0":{...},...), but you try to dereference is with ->[0] which is an array dereference.
You want:
$decoded->{list}{0}{weather}[0]{description}
instead of
$decoded->{list}[0]{weather}[0]{description}

What is "compressed JSON"?

I see a lot of references to "compressed JSON" when it comes to different serialization formats. What exactly is it? Is it just gzipped JSON or something else?
Compressed JSON removes the key:value pair of json's encoding to store keys and values in seperate parallel arrays:
// uncompressed
JSON = {
data : [
{ field1 : 'data1', field2 : 'data2', field3 : 'data3' },
{ field1 : 'data4', field2 : 'data5', field3 : 'data6' },
.....
]
};
//compressed
JSON = {
data : [ 'data1','data2','data3','data4','data5','data6' ],
keys : [ 'field1', 'field2', 'field3' ]
};
This method of usage i found here
Content from link (http://www.nwhite.net/?p=242)
rarely find myself in a place where I am writing javascript applications that use AJAX in its pure form. I have long abandoned the ‘X’ and replaced it with ‘J’ (JSON). When working with Javascript, it just makes sense to return JSON. Smaller footprint, easier parsing and an easier structure are all advantages I have gained since using JSON.
In a recent project I found myself unhappy with the large size of my result sets. The data I was returning was tabular data, in the form of objects for each row. I was returning a result set of 50, with 19 fields each. What I realized is if I augment my result set I could get a form of compression.
// uncompressed
JSON = {
data : [
{ field1 : 'data1', field2 : 'data2', field3 : 'data3' },
{ field1 : 'data4', field2 : 'data5', field3 : 'data6' },
.....
]
};
//compressed
JSON = {
data : [ 'data1','data2','data3','data4','data5','data6' ],
keys : [ 'field1', 'field2', 'field3' ]
};
I merged all my values into a single array and store all my fields in a separate array. Returning a key value pair for each result cost me 8800 byte (8.6kb). Ripping the fields out and putting them in a separate array cost me 186 bytes. Total savings 8.4kb.
Now I have a much more compressed JSON file, but the structure is different and now harder to work with. So I implement a solution in Mootools to make the decompression transparent.
Request.JSON.extend({
options : {
inflate : []
}
});
Request.JSON.implement({
success : function(text){
this.response.json = JSON.decode(text, this.options.secure);
if(this.options.inflate.length){
this.options.inflate.each(function(rule){
var ret = ($defined(rule.store)) ? this.response.json[rule.store] : this.response.json[rule.data];
ret = this.expandData(this.response.json[rule.data], this.response.json[rule.keys]);
},this);
}
this.onSuccess(this.response.json, text);
},
expandData : function(data,keys){
var arr = [];
var len = data.length; var klen = keys.length;
var start = 0; var stop = klen;
while(stop < len){
arr.push( data.slice(start,stop).associate(keys) );
start = stop; stop += klen;
}
return arr;
}
});
Request.JSON now has an inflate option. You can inflate multiple segments of your JSON object if you so desire.
Usage:
new Request.JSON({
url : 'url',
inflate : [{ 'keys' : 'fields', 'data' : 'data' }]
onComplete : function(json){}
});
Pass as many inflate objects as you like to the option inflate array. It has an optional property called ’store’ If set the inflated data set will be stored in that key instead.
The ‘keys’ and ‘fields’ expect strings to match a location in the root of your JSON object.
Based in Paniyar's answer, we can convert a List of Objects in "compressed" Json format using C# like this:
var JsonString = serializer.Serialize(
new
{
cols = new[] { "field1", "field2", "field3"},
items = data.Select(x => new object[] {x.field1, x.field2, x.field3})
});
I used an array of object because the fields can be int, bool, string...
More Reduction:
If the field is repeated very often and it is a string type, you can get compressed a little be more if you add a distinct list of that field... for instance, a field name job position, city, etc are excellent candidate for this. You can add a distinct list of this items and in each item change the value for a reference number. That will make your Json more lite.
Compressed:
[["KeyA", "KeyB", "KeyC", "KeyD", "KeyE", "KeyF"],
["ValA1", "ValB1", "ValC1", "ValD1", "ValE1", "ValF1"],
["ValA2", "ValB2", "ValC2", "ValD2", "ValE2", "ValF2"],
["ValA3", "ValB3", "ValC3", "ValD3", "ValE3", "ValF3"],
["ValA4", "ValB4", "ValC4", "ValD4", "ValE4", "ValF4"]]
Uncompressed:
[{KeyA: "ValA1", KeyB: "ValB1", KeyC: "ValC1", KeyD: "ValD1", KeyE: "ValE1", KeyF: "ValF1"},
{KeyA: "ValA2", KeyB: "ValB2", KeyC: "ValC2", KeyD: "ValD2", KeyE: "ValE2", KeyF: "ValF2"},
{KeyA: "ValA3", KeyB: "ValB3", KeyC: "ValC3", KeyD: "ValD3", KeyE: "ValE3", KeyF: "ValF3"},
{KeyA: "ValA4", KeyB: "ValB4", KeyC: "ValC4", KeyD: "ValD4", KeyE: "ValE4", KeyF: "ValF4"}]
The most likely answer is that it really is just gzipped JSON. There is no other standard meaning to this phrase.
Re-organizing a homogenous array of JSON objects into a pair of arrays is a very useful technique to make the payload smaller and to speed up encoding and decoding, it is not commonly called "compressed JSON". I haven't run across it ever in open source or any open API, but we use this technique internally and call it "jsontable".