How to parse this JSON object/string? - json

I am trying to parse the JSON written # http://a0.awsstatic.com/pricing/1/ec2/sles-od.min.js
Here is a quick snippet from above link:
{vers:0.01,config:{rate:"perhr",valueColumns:["vCPU","ECU","memoryGiB","storageGB","sles"],currencies:["USD"],regions:[{region:"us-east",instanceTypes:[{type:"generalCurrentGen",sizes:[{size:"t2.micro",vCPU:"1",ECU:"variable",
...
...
...
...
Please visit the aforementioned link to see the complete JSON.
As seen above, none of the keys of above JSON have Double Quotes around them.
This leads to malformed JSON string and my JSON parser is failing at it. I also tried putting this JSON in http://www.jsoneditoronline.org/ and it fails as well.
Now, this is the same link which is used by Amazon to display various prices of their EC2 instance. So I think I am missing something here. My Googling led me to believe that above thing is not JSON and is instead JSONP.. I don't understand what is that.
Could you help me understand how to parse this JSON. BTW, I am doing this work in perl using JSON Module.
Some background:
Amazon Web Services does not have an API to get Pricing info programmatically. Hence I am parsing these links which is what amazon is doing while displaying pricing information here. Besides, I am not from programming space and perl is all I know.

Like you said JSONP or "JSON with padding" can't be parsed by json parser because it is not json (it is a different format). But it is actually a json with the prefix (padding)
The padding is typically the name of a callback function that wraps json.
In this case, its default callback names 'callback' and we can do a bit hackiest way by using Regular Expression to capture json that is wrapped by 'callback()' like this
s/callback\((.*)\);$/$1/s;
Also, if you would like to use JSON library, you can enable allow_barekey which means you don't need those quotes around those keys.
Below is my working code. I use LWP::Simple to get the content for the given and Data::Dump to print the isolated data structure.
use strict;
use warnings;
use LWP::Simple;
use JSON;
my $jsonp = get("http://a0.awsstatic.com/pricing/1/ec2/sles-od.min.js")
or die "Couldn't get url";
( my $json = $jsonp ) =~ s/callback\((.*)\);$/$1/s; #grap the json from $jsonp and store in $json variable
my $hash = JSON->new->allow_barekey->decode ( $json );
use Data::Dump;
dd $hash;
Outputs:
{
config => {
currencies => ["USD"],
rate => "perhr",
regions => [
{
instanceTypes => [
{
sizes => [
{
ECU => "variable",
memoryGiB => 1,
size => "t2.micro",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.023 } }],
vCPU => 1,
},
{
ECU => "variable",
memoryGiB => 2,
size => "t2.small",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.056 } }],
vCPU => 1,
},
{
ECU => "variable",
memoryGiB => 4,
size => "t2.medium",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.152 } }],
vCPU => 2,
},
{
ECU => 3,
memoryGiB => 3.75,
size => "m3.medium",
storageGB => "1 x 4 SSD",
valueColumns => [{ name => "os", prices => { USD => "0.170" } }],
vCPU => 1,
},
....

As said in comments above, it is not JSON so it can't be parsed by JSON parser... But for an quick & (very)dirty work, you can try the JSON::DWIW module.
The next code:
use 5.014;
use warnings;
use WWW::Mechanize;
use Data::Dump;
use JSON::DWIW;
my $mech = WWW::Mechanize->new();
my $jsonstr = $mech->get('http://a0.awsstatic.com/pricing/1/ec2/sles-od.min.js')->content;
($jsonstr) = $jsonstr =~ /callback\((.*)\)/s;
my $json_obj = JSON::DWIW->new;
my $data = $json_obj->from_json( $jsonstr );
dd $data;
prints a structure what maybe is what you want, e.g.:
{
config => {
currencies => ["USD"],
rate => "perhr",
regions => [
{
instanceTypes => [
{
sizes => [
{
ECU => "variable",
memoryGiB => 1,
size => "t2.micro",
storageGB => "ebsonly",
valueColumns => [{ name => "os", prices => { USD => 0.023 } }],
vCPU => 1,
},
{

Related

How to unit test GraphQL responses on Lumen?

I'm trying to test an API I built with Lumen (PHP), but I'm stuck on unit test my GraphQL responses`.
This is what I have tried:
class MovieQueryTest extends Tests\GraphQLTestCase
{
use DatabaseMigrations;
public function testCanSearch()
{
Movie::create([
'name' => 'Fast & Furious 8',
'alias' => 'Fast and Furious 8',
'year' => 2016
]);
$response = $this->post('/graphql/v1', [
'query' => '{movies(search: "Fast & Furious"){data{name}}}'
]);
$response->seeJson([
'data' => [
'movies' => [
'data' => [
'name' => 'Fast & Furious 8'
]
]
]
]);
}
}
This is what I got:
PHPUnit 7.5.6 by Sebastian Bergmann and contributors.
F..... 6
/ 6 (100%)
Time: 690 ms, Memory: 24.00 MB
There was 1 failure:
1) MovieQueryTest::testCanSearch Unable to find JSON fragment
["data":{"movies":{"data":{"name":"Fast & Furious"}}}] within
[{"data":{"movies":{"data":[]}}}]. Failed asserting that false is
true.
The problem is that my data structure doesn't match the JSON's structure. While my data is inside an Array, the JSON's data is inside an Object and I can't figure out how to make it match:
["data":{"movies":{"data":{"name":"Fast & Furious 8"}}}]
[{"data":{"movies":{"data":[{"name":"Fast & Furious 8"}]}}}]
How can I make my data structure match the JSON's data structure or there is a better way to unit test GraphQL responses on Lumen?
You need to wrap 'name' => 'Fast & Furious 8' inside it's own array, for example:
The following:
$array = [
'data' => [
'movies' => [
'data' => [
['name' => 'Fast & Furious 8']
]
]
]
];
Should output:
{"data":{"movies":{"data":[{"name":"Fast & Furious 8"}]}}}

Getting error while using GroupBy and Pagination in Eloquent

I'm trying to use eloquent to get me a grouped by response and at the same time give me a Pagination response (The one that gives me the link to the second page).
I'm trying to do this:
App\Eating::Where('student_id', 2)->orderBy('created_at', 'DESC')->groupBy(function ($row) {
return Carbon\Carbon::parse($row->created_at)->format('Y-m-d');
})->paginate(25);
But, I'm getting this error when running it in the Tinker:
PHP warning: strtolower() expects parameter 1 to be string, object given in D:\Folder\vendor\laravel\framework\src\Illuminate\Database\Grammar.php on line 58
without the groupBy, I'm getting the correct result:
>>> App\Eating::Where('student_id', 2)->orderBy('created_at', 'DESC')->paginate(25)->toArray();
=> [
"total" => 1,
"per_page" => 25,
"current_page" => 1,
"last_page" => 1,
"next_page_url" => null,
"prev_page_url" => null,
"from" => 1,
"to" => 3,
"data" => [
[
"id" => 5,
"status" => "Comeu Bem",
"created_at" => "2017-07-05 13:55:25",
"updated_at" => "2017-07-05 13:55:25",
],
],
]
BUT, when I remove the pagination, I do get the error but only because I added the get():
>>> App\Eating::Where('student_id', 2)->orderBy('created_at', 'DESC')->groupBy(function ($row) {
... return Carbon\Carbon::parse($row->created_at)->format('Y-m-d');
... })->get();
PHP warning: strtolower() expects parameter 1 to be string, object given in D:\Joao\git\F1Softwares\Code\Server\F1Softwares\vendor\laravel\framework\src\Illuminate\Database\Grammar.php on line 58
>>>
>>>
>>> App\Eating::Where('student_id', 2)->orderBy('created_at', 'DESC')->groupBy(function ($row) {
... return Carbon\Carbon::parse($row->created_at)->format('Y-m-d');
... });
=> Illuminate\Database\Eloquent\Builder {#855}
Any idea what I could be doing wrong? I do need to have the orderBy AND the pagination, to make it easier for the app to show the results(It is a RestFul call).
Thanks,
João
You must call the groupBy() method on a collection, but it seems this won't work with paginate(). You could try using the forPage() method on the collection:
App\Eating::where('student_id', 2)->orderBy('created_at', 'DESC')
->get()->groupBy(function ($eating) {
return $eating->created_at->format('Y-m-d');
})->forPage(1, 25);
Also, just a note, you don't need to use Carbon to parse the date, Eloquent does this for you.
Alternatively, you could try to manually create your paginator once you have the collection grouped using Illuminate\Pagination\LengthAwarePaginator.
$eatings = App\Eating::where('student_id', 2)->orderBy('created_at', 'DESC')
->get()->groupBy(function ($eating) {
return $eating->created_at->format('Y-m-d');
});
$paginatedEatings = new LengthAwarePaginator($eatings, $eatings->count(), 25);
return $paginatedEatings->toArray();

Need help in identifying TYPE of document ingested by Elasticsearch (through Logstash)

I used Logstash to ingest csv files from https://www.kaggle.com/wcukierski/the-simpsons-by-the-data and saved it to Elasticsearch. For starters, I ingested simpsons_characters.csv using the following conf:
input {
file {
path => "/Users/xyz/Downloads/the-simpsons-by-the-data/simpsons_characters.csv"
start_position => beginning
sincedb_path => "/dev/null"
}
}
filter {
csv {
columns => ["id", "name", "normalized_name", "gender"]
separator => ","
}
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
}
}
However, when I query like so: http://localhost:9200/simpsons/name/Lou
where
simpsons = index
name = type (I think ... not sure)
I get the following response back:
{
"_index": "simpsons",
"_type": "name",
"_id": "Lou",
"found": false
}
So, the question is, why am I not getting the correct response. Further, when you do bulk ingestion through csv, what is the type of the document?
Thanks!
The default type in Logstash Elasticsearch output is logs. So, no matter how you define your IDs (either take it from the csv - document_id => "%{id}" or let ES define its own), you can get those documents as http://localhost:9200/simpsons/logs/THE_ID.
If you don't know the id and want to simply check if something is there: http://localhost:9200/simpsons/logs/_search?pretty.
If you want to see what is the mapping of your index, for example to find out the _type of the index: http://localhost:9200/simpsons/_mapping?pretty.
To change the default _type:
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
document_type => "characters"
document_id => "%{id}"
}
Here you haven't specified id field in your logstash output. In this case elasticsearch would asign a random id to your documents and you are searching for a document with id=Lou.
Adding document_id => "%{id}" would solve your problem.
output {
stdout {
codec => rubydebug
}
elasticsearch {
hosts => "localhost"
action => "index"
index => "simpsons"
document_id => "%{id}"
}
}

Perl Catalyst - Couldn't render template..........not found

The error I am getting in the development server:
[info] *** Request 2 (0.000/s) [681] [Thu Dec 12 21:05:39 2013] ***
[debug] Path is "homescreen"
[debug] "GET" request for "homescreen" from "192.168.1.100"
[debug] Rendering template "homescreen/homescreen.tt2"
[error] Couldn't render template "homescreen/homescreen.tt2: file error - homescreen/homescreen.tt2: not found"
[error] Couldn't render template "homescreen/homescreen.tt2: file error - homescreen/homescreen.tt2: not found"
[debug] Response Code: 500; Content-Type: text/html; charset=utf-8; Content-Length: 14312
[info] Request took 0.033915s (29.485/s)
.------------------------------------------------------------+-----------.
| Action | Time |
+------------------------------------------------------------+-----------+
| /homescreen | 0.000341s |
| /end | 0.014055s |
| -> Myproject::View::HTML->process | 0.013049s |
'------------------------------------------------------------+-----------'
What I am doing:
I have the following Controller/Homescreen.pm:
package Myproject::Controller::Homescreen;
use strict;
use warnings;
use parent 'Catalyst::Controller';
use Data::Dumper;
use JSON;
__PACKAGE__->config->{namespace} = '';
sub homescreen :Path('/homescreen') :Args(0) {
my ( $self, $c ) = #_;
print STDERR "IN THE HOMESCREEN ACTION\n";
$c->stash({template => 'homescreen/homescreen.tt2',
title => 'Home Screen'
});
}
I have the following View/HTML.pm:
package Myproject::View::HTML;
use Moose;
use namespace::autoclean;
extends 'Catalyst::View::TT';
__PACKAGE__->config({
#Changed default TT extension to TT2
TEMPLATE_EXTENSION => '.tt2',
render_die => 1,
});
I have the following lib/Myproject.pm:
__PACKAGE__->config(
name => 'Myproject',
# Disable deprecated behavior needed by old applications
disable_component_resolution_regex_fallback => 1,
#enable_catalyst_header => 1, # Send X-Catalyst header
);
__PACKAGE__->config(
#Configure the view
'View::HMTL' => {
#Set the location for TT files
INCLUDE_PATH => [
__PACKAGE__->path_to( 'root', 'src' ),
],
},
);
# Start the application
__PACKAGE__->setup();
I then have a root/src/homescreen/homescreen.tt2 withing my Catalyst directory that contains all my html code (eventually it will use the template toolkit,but at the moment it is purely html and javscript code which I know is fine).
The error I get on the application page in my browser is:
Couldn't render template "homescreen/homescreen.tt2: file error - homescreen/homescreen.tt2: not found"
I have tried using DEBUG => 'undef' in my HTML.pm View to help with debugging, but I don't seem to get any extra output.
There is probably something very obvious I am overlooking but I cannot work out what it is.
Update
I have just noticed the following in the Config section of my browser debug screen:
Config
do {
my $a = {
"Action::RenderView" => {
ignore_classes => [
"DBIx::Class::ResultSource::Table",
"DBIx::Class::ResultSourceHandle",
"DateTime",
],
scrubber_func => sub { ... },
},
"disable_component_resolution_regex_fallback" => 1,
"home" => "/home/fred/Myproject",
"name" => "Myproject",
"Plugin::ConfigLoader" => {},
"Plugin::Static::Simple" => {
debug => 1,
dirs => [],
ignore_dirs => [],
ignore_extensions => ["tmpl", "tt", "tt2", "html", "xhtml"], <---- IS THIS SIGNIFICANT AT ALL?
include_path => [
bless({
dirs => ["", "home", "fred", "Myproject", "root"],
file_spec_class => undef,
volume => "",
}, "Path::Class::Dir"),
],
mime_types => {},
mime_types_obj => bless({}, "MIME::Types"),
no_logs => 1,
},
"root" => 'fix',
"stacktrace" => { context => 3, verbose => 0 },
"static" => 'fix',
"View::HMTL" => {
INCLUDE_PATH => [
bless({
dirs => ["", "home", "fred", "Myproject", "root", "src"],
file_spec_class => undef,
volume => "",
}, "Path::Class::Dir"),
],
},
};
$a->{"root"} = $a->{"Plugin::Static::Simple"}{include_path}[0];
$a->{"static"} = $a->{"Plugin::Static::Simple"};
$a;
}
I take it this means it is ignoring my template file because it has the .tt2 file extension?
However, I am not setting this ignore_extensions attribute anywhere in my Catalyst project? Is this the cause of my problem or something totally unrelated?
It looks like your configuration isn't taking effect. Try putting your template in root/homescreen/homescreen.tt2 instead of root/src/homescreen/homescreen.tt2, and Catalyst finds it.
Ahh, you have a typo in your lib/Myproject.pm:
__PACKAGE__->config(
#Configure the view
'View::HMTL' => {
Try 'View::HTML' instead (notice you have HMTL - wrong spelling).

How do I get JSON output in PERL without \n but in a readable format?

I need to get a readable JSON object from a PERL script but it's not in a readable format.
This is the code that produces the JSON.
while (my ($orderID, $possessorName, $itemDescription, $customerPickUpTime, $customerDropOffTime, $paymentAmount, $originAddress1, $originAddress2, $originNeighborhood, $originZipCode, $destinationAddress1, $destinationAddress2, $destinationNeighborhood, $destinationZipCode) = $sth->fetchrow_array)
{
%data = (orderID => $orderID, possessorName => $possessorName, itemDescription => $itemDescription, customerPickUpTime => $customerPickUpTime, customerDropOffTime => $customerDropOffTime, paymentAmount => $paymentAmount, originAddress1 => $originAddress1, originAddress2 => $originAddress2, originNeighborhood => $originNeighborhood, originZipCode => $originZipCode, destinationAddress1 => $destinationAddress1, destinationAddress2 => $destinationAddress2, destinationNeighborhood => $destinationNeighborhood, destinationZipCode => $destinationZipCode);
$json_obj = JSON->new->allow_nonref;
my $json_text = $json_obj->pretty->encode(\%data);
$query_results{"job$index"} = {"data" => $json_text};
$index++;
}
return $json_obj->pretty->encode(\%query_results, {ascii => 1, pretty => 1});
Everything works except when I look into the file (here's the printing line):
open (resultsFile, ">", "json_file.txt") || die "This doesn't work.";
print resultsFile "Results: \n\n $results";
The results are as follows:
{
"job3" : {
"data" : "{\n \"originAddress1\" : \"101 East 105th Street\",\n \"destinationZipCode\" : \"10128\",\n \"destinationNeighborhood\" : \"Upper East Side\",\n \"customerDropOffTime\" : \"2013-01-22 23:41:37\",\n \"originAddress2\" : \"\",\n \"paymentAmount\" : \"19.00\",\n \"customerPickUpTime\" : \"2013-01-22 22:56:37\",\n \"itemDescription\" : \"body\",\n \"destinationAddress1\" : \"180 East 93rd Street\",\n \"destinationAddress2\" : \"\",\n \"possessorName\" : \"Lisa Howard\",\n \"originZipCode\" : \"10029\",\n \"originNeighborhood\" : \"East Harlem\",\n \"orderID\" : \"723\"\n}\n"
},
The JSON object is formed correctly but the \n is the problem. It's not outputting with an actual newline. That's the issue.
This is because, $json_text is a string and not a hash. If you want to encode the whole thing as JSON, you must create an appropriate data structure
$query_results{"job$index"} = {"data" => \%data};
and give that as a whole to encode.