How to POST json file using REST API in Perl? - json

There is an application that generates multiple records per day, containing different types of attributes like EmpName, EmpDesig etc. and stores it in JSON format. This application then needs to make a call towards external REST api to POST this data that is in JSON format. The external application will read the JSON file, parse it, and store each record in the PostgreSQL database. How do I write a REST api in Perl for this requirement? Do I need any Perl framework like Mojolicious or Catalyst or Perl modules like JSON, Rest::Client are enough for this requirement? Please suggest.

If you do want to use a framework, it'll simplify your code a lot.
You could do it with the Dancer framework in a single line:
perl -e 'use Dancer; any "/", sub{ warn to_dumper from_json request->body; return "ok" }; start'
However, this answer is just a start, it's not robust nor contains any checks.

You don't strictly need a framework. It's easiest to get started with Plack::Request. The next step up would be adding a router like Router::Resource. Upgrading to a framework makes sense when you find this style of programming too tedious.
use Plack::Request qw();
use HTTP::Status qw(
HTTP_NO_CONTENT HTTP_NOT_FOUND HTTP_METHOD_NOT_ALLOWED
HTTP_UNSUPPORTED_MEDIA_TYPE HTTP_UNPROCESSABLE_ENTITY
HTTP_INTERNAL_SERVER_ERROR
);
use JSON::MaybeXS qw(decode_json);
use Syntax::Keyword::Try;
require IO::Handle;
use DBI qw();
require DBD::Pg;
my $app = sub {
my ($env) = #_;
my $req = Plack::Request->new($env);
return $req->new_response(HTTP_NOT_FOUND)->finalize
unless '/receive_json' eq $req->path_info;
return $req->new_response(HTTP_METHOD_NOT_ALLOWED)->finalize
unless 'POST' eq $req->method;
return $req->new_response(HTTP_UNSUPPORTED_MEDIA_TYPE)->finalize
unless $req->content_type =~ m'^application/.*json$';
my $body = do {
IO::Handle->input_record_separator(undef);
$req->input->getline;
};
my $json;
try {
$json = decode_json $body;
} catch {
warn "could not decode JSON: $#";
return $req->new_response(HTTP_UNPROCESSABLE_ENTITY)->finalize;
}
my $dbh;
try {
$dbh = DBI->connect(
"dbi:Pg:dbname=$ENV{PGDATABASE}",
$ENV{PGUSER},
$ENV{PGPASSWORD},
{pg_enable_utf8 => 1, RaiseError => 1}
);
} catch {
warn "could not connect to database: $#";
return $req->new_response(HTTP_INTERNAL_SERVER_ERROR)->finalize;
}
my $sth = $dbh->prepare(
'insert into sometable(EmpName, EmpDesig) values (?,?)'
);
try {
$sth->execute($json->{EmpName}, $json->{EmpDesig});
} catch {
warn "could not insert: $#";
return $req->new_response(HTTP_INTERNAL_SERVER_ERROR)->finalize;
}
return $req->new_response(HTTP_NO_CONTENT)->finalize;
};

Related

JSON decode failed

Following is the code snippet where I am observing error: "malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at"
Error observed is at the decode_json line. Can someone point out what is the error?
my $serverurl = "http://mycompany.net/rest/api/2/";
my $username = 'my.email#domain.com';
my $password = "mypassword\#2019";
my $i ;
my $test;
my $headers = {Accept => 'application/json', Authorization => 'Basic ' .encode_base64($username . ':' . $password)};
my $client = REST::Client->new();
my $idartinstance;
my $idartinstance1;
if (!$idartinstance)
{
print " Trying to Connect to URL using REST client interface \n\n";
$idartinstance1 = $client->GET($serverurl."serverinfo",$headers);
$idartinstance = decode_json($idartinstance1->responseContent());
}
When I print $idartinstance, I get this:
REST::Client=HASH(0x8682024)->responseContent()
Does this mean, it is not able to find REST client?
[EDIT] I have modified the script as below and no difference in the errors.
my $serverurl = "https://mycompany.net/rest/api/3/";
my $username = 'my.email#domain.com';
my $password = 'pf9fCdkGXmi4pMHiwIh74A0D';
my $headers = {Accept => 'application/json', Authorization => 'Basic ' . encode_base64($username . ':' . $password)};
my $client = REST::Client->new();
if (!$idartinstance)
{
print " Trying to Connect to JIRA using REST client interface \n\n";
$client->GET($serverurl."serverInfo", $headers);
print $client->responseContent();
$idartinstance = decode_json($client->responseContent());
}
Now I have used encoded password. Error is same: malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)"). Tried accessing "https://mycompany.net/rest/api/3/serverInfo" via web browser and able to get the details.
Once you get a response, you have to check that its what you want.
if( $client->responseCode() eq '200' ){
print "Success\n";
}
You may also want to check that the content-type is what you expect. If it's supposed to be JSON, check that it is:
if( $client->responseHeader('Content-Type') =~ m|\Aapplication/json\b| ) {
print "Got JSON\n";
}
Once you've established that you have what you wanted, pass the message body off to the JSON decoder.
my $data = decode_json($client->responseContent());
You might also try to catch errors where you should have valid JSON but don't. The block eval can handle that (and see the many sources of the proper employment of eval for its best use):
my $data = eval { decode_json(...) };
I find that I tend to get the wrong content in two situations:
the wrong endpoint, from which a 404 handler returns HTML
a captive portal, which also returns HTML
I think you're misreading the documentation for the module. From the synopsis there, the example that most closely resembles yours is the first one:
my $client = REST::Client->new();
$client->GET('http://example.com/dir/file.xml');
print $client->responseContent();
Notice, in particular, that this example does nothing with the return value from GET(). In your example you do the equivalent of this:
my $client = REST::Client->new();
my $resp = $client->GET('http://example.com/dir/file.xml');
print $resp->responseContent();
As it happens, although there is no documented return value from GET() [Update: I was wrong here - see the first comment - but the return value is really only intended for chaining method calls], it actually returns the object that it was passed, so your approach should work. But it's generally a bad idea to not follow the documentation.
So what is actually going wrong? Well, I'm not sure. As I said, your approach should (accidentally) work. But the error message you're getting tells us that what you're passing to decode_json() is a REST::Client object, not a string containing JSON. I don't think that's how your code should work. Perhaps the code you've shown us isn't actually the code you're running.
The best approach to debug this is to follow the advice from Quentin in the first comment on your question - print the value that you're trying to pass to decode_json() before passing it to the function. In fact, that's good general programming advice - originally write out your code step by step, and only combine steps once you know that the individual steps are working correctly.
Using your variable names, I think your code should look like this:
my $client = REST::Client->new();
# ...other code...
$client->GET($serverurl."serverinfo", $headers);
print $client->responseContent();
# And, only once you've established that
# $client->responseContent() returns what
# you expect, you can add this:
$idartinstance = decode_json($client->responseContent());
If the print() statement doesn't show you JSON, then update your question to add whatever is printed and we'll take a further look.

GuzzleHttp - Get JSON encoded body

I created an API client for my company to fetch orders from our distributors. I need to acknowledge download of orders back to them with a PUT. The PUT is working properly but I get an error on their confirmation of my acknowledgement.
Using Postman, I get a JSON body message back.
When I PUT a acknowledgement back, I get the following error:
Type error: Argument 1 passed to GuzzleHttp\Client::send() must
implement interface Psr\Http\Message\RequestInterface, instance of
GuzzleHttp\Psr7\Response given, called in
/var/www/orders/app/Http/Controllers/edi/OrderController.php on line 86
This is line 86:
$response = $client->send($apirequest);
The relevant code:
use Illuminate\Http\Request;
use App\Http\Requests;
use App\Http\Controllers\Controller;
use GuzzleHttp\Exception\GuzzleException;
use GuzzleHttp\Client as GuzzleHttpClient;
use GuzzleHttp\Psr7\Stream;
use Illuminate\Support\Facades\Input;
use Response;
use XmlParser;
use Psr\Http\Message\RequestInterface;
public function orderConfirm()
{
$uri = config('services.orders.orderack');
$formdata = Input::all();
$orders = Input::get('orders');
try {
$client = new GuzzleHttpClient([
'headers'=> [
'Authorization' => '$user',
'ContractID' => '$contract',
'Content-Type' => 'application/json']
]);
$apirequest = $client->request('PUT', $uri,
['body' => json_encode(
[
$orders
]
)]
);
$response = $client->send($apirequest);
$contents = (string) $response->getBody();
return $contents;
}
catch (RequestException $ex) {
//Exception Handling
echo $ex;
}
Output from Postman was:
"Number of Orders Acknowledged: 1"
from other posts on SO, this:
$contents = (string) $response->getBody();
is the way to get the body and other people fixed their problems, but it's not working for me.
Obviously I'm still missing something here!
Calling $client->request() actually does the request (which is why it's returning an instance of GuzzleHttp\Psr7\Response) instead of building a request object to send later. You don't need to tell the client to send anything because it's already been sent; you just need to set the $response variable to the value of the call to $client->request().
This can be seen in the Body example in their PSR7 documentation.
$response = $client->request('GET', 'http://httpbin.org/get');
To build a request object manually, you will have to create an instance of GuzzleHttp\Psr7\Request using its constructor, as documented under Requests.
// Create a request using a completely custom HTTP method
$request = new \GuzzleHttp\Psr7\Request('MOVE', 'http://httpbin.org/move');
echo $request->getMethod();
// MOVE

Fetching JSON in Perl

I've been working building an emulator in Perl and one of the issues I'm facing is parsing JSON files located in the computer. When I try fetching them from my server, they work fine...
method getContent(\#arrURLS) {
my %arrInfo;
my $resUserAgent = Mojo::UserAgent->new;
foreach my $strURL (#arrURLS) {
$resUserAgent->get($strURL => sub {
my($resUserAgent, $tx) = #_;
if ($tx->success) {
my $strName = basename($strURL, '.json');
my $arrData = $tx->res->body;
$arrInfo{$strName} = $arrData;
}
Mojo::IOLoop->stop;
});
Mojo::IOLoop->start;
}
return \%arrInfo;
}
Let's assume #arrURLS is:
my #arrURLS = ("file:///C:/Users/Test/Desktop/JSONS/first.json", "file:///C:/Users/Test/Desktop/JSONS/second.json");
The above url's are the one's that aren't working, however if I change that to:
my #arrURLS = ("http://127.0.0.1/test/json/first.json", "http://127.0.0.1/test/json/second.json");
it works.
Also I would like to use something better than Mojo::UserAgent because it seems a bit slow, when I was using Coro with LWP::Simple it was much faster but unfortunately Coro is broken in Perl 5.22...
User Agents are mainly for downloading files through http. They are usually not expected to handle filesystem URIs. You need to open and read the file yourself, or use a module like File::Slurp that does it for you.
It could look something like this.
use File::Slurp 'read_file';
method getContent(\#arrURLS) {
my %arrInfo;
my $resUserAgent = Mojo::UserAgent->new;
foreach my $strURL (#arrURLS) {
if (substr($strURL, 0, 4) eq 'file') {
$arrInfo{basename($strURL, '.json')} = read_file($strURL);
} else {
$resUserAgent->get($strURL => sub {
my($resUserAgent, $tx) = #_;
if ($tx->success) {
my $strName = basename($strURL, '.json');
my $arrData = $tx->res->body;
$arrInfo{$strName} = $arrData;
}
Mojo::IOLoop->stop;
});
Mojo::IOLoop->start;
}
}
return \%arrInfo;
}
Myself using WWW::Mechanize for all such tasks. From the doc:
WWW::Mechanize is a proper subclass of LWP::UserAgent and you can also
use any of LWP::UserAgent's methods.
what means you can feed it with file:// type URLs too.
For example, the following one-liner dumps your passwd file.
perl -MWWW::Mechanize -E 'say WWW::Mechanize->new->get("file://etc/passwd")->content'
or an example without any error handling...
use 5.014;
use warnings;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new;
$mech->get('file://some/path');
say $mech->content;
Anyway, probably is better to use for local files some file-based utility, myself using for all file-things the Path::Tiny module, which has (not limited only) an method for file slurping, such:
use Path::Tiny;
my $content = path('/some/path')->slurp;
or just plain perl:
open my $fh, '<', '/some/file' or die "...";
my $content = do { local $/; <$fh> };
close $fh;
It's important always to say what additional modules you're using. I think your code uses Method::Signatures, and I've tested the code below only to check that it compiles with that module in place
Mojolicious is an excellent tool for its purpose, but it is focused on HTTP URLs. LWP::UserAgent is much more general-purpose, and the documentation for LWP says this
Provides an object oriented model of HTTP-style communication. Within this framework we currently support access to http, https, gopher, ftp, news, file, and mailto resources
Your method becomes something like this. It's untested
method get_content(\#urls) {
my %info;
my $ua = LWP::UserAgent->new;
for my $url (#urls) {
my $res = $ua->get($url);
die $res->status_line unless $res->is_success;
my $name = basename($url) . '.json';
my $data = $res->decoded_content;
$info{$name} = $data;
}
\%info;
}
I would also encourage you to drop Hungarian notation in the context of Perl code, as the language already has its sigils that denote the data type
#arrURLS duplicates the information that this arrURLS is an array, while %arrInfo is just wrong as this arrInfo is a hash. $arrData is actually a scalar, although perhaps some indicator that it is also a reference may help, and $arrURLS[0] is also a scalar (hence the dollar)
There is also nothing to stop you using $arrURLS (which is a completely separate variable from #arrURLS)

JSON feeds context within Drupal 7

I'm trying to connect to a REST web service with Drupal 7. Data is supplied via a url and I want to pull it into the D7 site to create and populate nodes. I'm using the feeds module to map the content and the JSONPath parser as the parser. I have also included Feeds HTTPFetcher Append Headers module so that I can add custom headers in to make sure it is JSON being returned and not badly formatted xml which I was getting before.
Custom headers:
Accept|application/json
Content-Type|application/json
I have a url that looks like http://192.136.0.31:8080/places/getAll
Further to the above I have the Devel module in place which is returning the feed array in the page using the code below in my template.php:
$request = drupal_http_request('http://192.136.0.31:8080/places/getAll', $options);
dpm(drupal_json_decode($request->data));
This is what Devel returns:
... (Array, 1192 elements)
0 (Array, 17 elements)
Address1 (String, 19 characters ) 1-2 The Road name
Address2 (String, 9 characters ) Shuinnad
The problem I'm having is getting this into feeds and mapping to the relevant fields. I don't know what should go in the context field - I have tried $...*, $.*, $...[*] but no luck.
When I click on an Array element from the devel output it shows $...[0]['Address1'] which suggests that should be the context - no luck.
Quick Update - using the drupal_json_decode function I can split the array out how I need to using php
foreach ($json as $section => $items) {
foreach ($items as $key => $value) {
//if ($key == 'Address1') {
echo "$section:\t$key\t: $value<br>";
//}
// check whether the current item is an array!
if(is_array($value)) {
echo "$key is sub array:<br />";
foreach($value as $subKey => $subValue)
echo "$subKey:\t$subValue<br />";
}
}
}
The question still stands, how do I replicate that in Feeds using the JSON parser?
You may need to try this module: Feeds JSONPath Parser
See this tutorial
Context: is where you put the JSONPath expression that represents the path to the array representing your data. For instance, in my file, this would be $.albums.data.*. If this were a PHP array, that would be basically
foreach($facebook['albums']['data'] as $value);
See more at: http://glassdimly.com/blog/tech/drupal-7-feeds-extensible-parsers-jsonpath-map-json-fields/feeds-extensible-parsers#sthash.BrIutjfl.dpuf

Yii Restful API will create on POST but not populate any variables

Hello I am currently trying to use POSTMAN to test an early API that is based of of this post
http://www.yiiframework.com/wiki/175/how-to-create-a-rest-api/
I am having an error when trying to either submit variables via POST in php url style or even sending an object. The response from the API comes back 200 and it creates a new entry into the database, but unfortunately it will not take any info from the post variables or jason object, it just comes up null. It seems now that that code is just looking through $_POST variables to and trying to match them to a model variable and if so, it should update it save it, However when i try to send through url parameters in POSTMAN or even change content type json and send raw json object I seem to have no luck with it.
Also I really only need it to decode a jason object and not post parameters so maybe that is where I will start by removing the $post loop and working on retrieving a JSON object instead. Thanks for any help!
public function actionCreate()
{
switch($_GET['model'])
{
// Get an instance of the respective model
case 'event':
$model = new Event;
break;
case 'media':
$model = new Media;
break;
case 'comment':
$model = new Comment;
break;
default:
$this->_sendResponse(501,
sprintf('Mode <b>create</b> is not implemented for model <b>%s</b>',
$_GET['model']) );
Yii::app()->end();
}
// Try to assign POST values to attributes
foreach($_POST as $var=>$value) {
// Does the model have this attribute? If not raise an error
if($model->hasAttribute($var))
$model->$var = $value;
else
$this->_sendResponse(500,
sprintf('Parameter <b>%s</b> is not allowed for model <b>%s</b>', $var,
$_GET['model']) );
}
// Try to save the model
if($model->save())
$this->_sendResponse(200, CJSON::encode($model));
else {
// Errors occurred
$msg = "<h1>Error</h1>";
$msg .= sprintf("Couldn't create model <b>%s</b>", $_GET['model']);
$msg .= "<ul>";
foreach($model->errors as $attribute=>$attr_errors) {
$msg .= "<li>Attribute: $attribute</li>";
$msg .= "<ul>";
foreach($attr_errors as $attr_error)
$msg .= "<li>$attr_error</li>";
$msg .= "</ul>";
}
$msg .= "</ul>";
$this->_sendResponse(500, $msg );
}
}
Fixed by removing $POST loop and changing to a JSON object scheme only. Here is the code if anyone happens to find themselves in this situation.
//read the post input (use this technique if you have no post variable name):
$post = file_get_contents("php://input");
//decode json post input as php array:
$data = CJSON::decode($post, true);
//load json data into model:
$model->attributes = $data;
Used that instead of the foreach loop through $_POST variables. Now it accepts a json object instead. Happy Coding all!
Honestly, I'm not sure what your problem is. If you can see the values POSTed in $_POST but not assigned to $model, it's probably because you did not specify the validation rules for those fields in the model. If you do not need to validate the fields, simply mark them as 'safe' in the model's rules() function like below.
public function rules() {
return array(
array('fieldName1,fieldName2', 'safe'),
);
}