Scrape a particular area of site content With a Secure Login - html

I am trying to scrape some particular text of a website which is login secured
here is the tutorial on this using curl
http://www.digeratimarketing.co.uk/2008/12/16/curl-page-scraping-script/
But I am unable to implement this into my curl codes
here is my curl script
$url = "http://aftabcurrency.com/login_script.php";
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$cookie = 'cookies.txt';
$timeout = 30;
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout );
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch,CURLOPT_POSTFIELDS,"user_name=user&user_password=pass&passcode=code");
$result = curl_exec($ch);
curl_close($ch);
$source = $result;
if(preg_match("/(CC3300\">)(.*?)(<\/font>)/is",$source,$found)){
echo $found[2];
}else{
echo "Text not found.";
}
for example in aftabcurrency.com I only wish to scrap only "Our Services Matters!" (this text changes every day)

what I would do is to "cut out" a text between start and beginning... in the source the text is starting by a text color 613A75 and ands with the closing < /font> tag.. here is a regex solution:
$source = file_get_contents("http://aftabcurrency.com/index.php");
if(preg_match("/(613A75\">)(.*?)(<\/font>)/is",$source,$found)){
echo $found[2];
}else{
echo "Text not found.";
}
if you want to do this with your text inside member area, add my source here to your source and replace the $source = file_get_contents... with $source = $result
there is also other way to do this, DomDocument and xpath or simple strpos / strstr / substr php functions.

Related

json_decode doesn't work with string

With this code :
$url = 'https://www.xxxxxxx.com/api/v1/phone/?apikey=xxxxxxxxxx&id='.$id;
$cURL = curl_init();
curl_setopt($cURL, CURLOPT_URL, $url);
curl_setopt($cURL, CURLOPT_HTTPGET, true);
curl_setopt($cURL, CURLOPT_HTTPHEADER, array(
'Content-Type: application/json',
'Accept: application/json'
));
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($cURL);
curl_close($cURL);
echo $result;
var_dump ($result);
var_dump(json_decode($result, true));
I get that :
{"telephone":"0 811 228 338"}
string(32) "{"telephone":"X XXX XXX XXX"}"
NULL
I don't understand why json_decode doesn't work here?
Thanks for helping me
with the little info i got i believe its a json formating problem.
<?php
$json = '{"foo-bar": 12345}';
$obj = json_decode($json);
print $obj->{'foo-bar'}; // 12345
?>
the json quotes are not valid
http://php.net/manual/en/function.json-decode.php
try it in some validator as this one http://jsonlint.com/
The solution was that the response is in UTF-8 but not UTF-8 (SANS BOM).
It means that there is a invisible caractere on the beginning of the string (U+FEFF).
The solution : $result = substr($result, 3);
Thanks for your help

request a web service from laravel that returns a json object

I have this Url : https://ipfind.co/?ip=188.225.179.138&auth=dde7cf52-2294-47e3-adb2-1e559099527e
when I request it from browser it gives me this response:
{"ip_address":"188.225.179.138","country":"Palestinian Territory, Occupied","country_code":"PS","continent":"Asia","continent_code":"AS","city":"Ramallah","county":null,"region":null,"region_code":null,"timezone":"GMT+2","owner":"COOLNET NEW COMMUNICATION PROVIDER","longitude":35.2,"latitude":31.9}
how can I invoke this url from laravel?
note that I tried this code, but it didn't work.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://ipfind.co/?ip=188.225.179.138&auth=dde7cf52-2294-47e3-adb2-1e559099527e');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec();
curl_close($ch);
dd($data);
Thanks in advance.
In the Curl you shall do this Easy 5 steps
Step 1
Initiate the Curl $cSession = curl_init();
Step 2
Define the Curl Url
curl_setopt($cSession,CURLOPT_URL,"https://ipfind.co/?ip=188.225.179.138&auth=dde7cf52-2294-47e3-adb2-1e559099527e");
curl_setopt($cSession,CURLOPT_RETURNTRANSFER,true);
curl_setopt($cSession,CURLOPT_HEADER, false);
Step 3
Execute the Curl
$result=curl_exec($cSession);
Step 4
Close the Curl Connection
curl_close($cSession);
Step 5
Print the Result
echo $result;
Putting all together
<?php
$cSession = curl_init();
curl_setopt($cSession,CURLOPT_URL,"https://ipfind.co/?ip=188.225.179.138&auth=dde7cf52-2294-47e3-adb2-1e559099527e");
curl_setopt($cSession,CURLOPT_RETURNTRANSFER,true);
curl_setopt($cSession,CURLOPT_HEADER, false);
$result=curl_exec($cSession);
curl_close($cSession);
echo $result;
?>
There is a PHP package for getting an IP address for IP Find that you can install with composer here that works great with Laravel.
https://github.com/tenfef/ipfind-php/

Documentation for Google FIT REST API dataset:aggregate

I am trying to get data from Google Fit using REST API. I am able to get individual data points for steps and calories through the documented REST APIs. However,I want them aggregated by day. I see the following resource in the Oauth2 playground:
https://www.googleapis.com/fitness/v1/users/{userId}/dataset:aggregate
However, I was not able to find any documentation on how to use this. Has anyone been successful in using this REST resource? Any pointers to the documentation / any examples would really help.
Thanks,
This is how i solved this REST call & its working code in my project.
$AccessToken = "TOKEN";
Data['aggregateBy'] =array(["dataTypeName"=>'com.google.step_count.delta',"dataSourceId"=>'derived:com.google.step_count.delta:com.google.android.gms:estimated_steps']);
$Data['bucketByTime']['durationMillis'] =86400000;
$Data['startTimeMillis'] =1487721600000;
$Data['endTimeMillis'] =1487772000000;
$JsonData = json_encode($Data);
$ApiUrl = 'https://www.googleapis.com/fitness/v1/users/me/dataset:aggregate?access_token='.$AccessToken;
$Ch = curl_init();
curl_setopt($Ch, CURLOPT_URL, $ApiUrl);
curl_setopt($Ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($Ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($Ch, CURLOPT_HEADER, 0);
curl_setopt($Ch, CURLOPT_HTTPHEADER, array('Content-type:application/json'));
curl_setopt($Ch, CURLOPT_POST, 1);
curl_setopt($Ch, CURLOPT_POSTFIELDS, $JsonData);
$Result = curl_exec($Ch);
curl_close($Ch);
print_r($Result);

curl request to google places api returns "bool(false)"

I've tried using the Javascript version of google places API, but now I'd like to move the code to server side with an API key. I've tried using all possible permutations and combinations of the curl request, but couldn't find a solution to it.
$url = urlencode('https://maps.googleapis.com/maps/api/place/autocomplete/json?input=' . $input . '&types=establishment&location=13.052415, 80.250824&radius=500&sensor=true&key=KEY');
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
$resp = curl_exec($curl);
curl_close($curl);
var_dump($resp);
NOTE: I've used my app's KEY in the "key" params.
Response I get => bool(false)
Any suggestions?

Post data using curl without showing the redirect URL in adress bar

Below is the my curl for senting some parameters to a url.But the parameter I m senting has sensitive datas like passwords and other things.But during posting the datas will be showed in the adress bar as redirect url.How to avoid this?.What to do to sent data without showing it in address bar?
$parameters='MerchantId='.$merchantId.'&Password='.$Password.'&ReferenceNo='.$ReferenceNo.'&RemoteIP='.$RemoteIP.'&Amount='.$Amount.'&BankId='.$BankId.'&Checksum='.$checksum.'&Name='.$Name.'&MobileNo='.$MobileNo.'&Email='.$Email;
//set POST variables
$url = 'https://payment.essecom.com/NetBanking/PayDirekt.jsp?'.$parameters;
//header('Location: '.$url);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-type: text/plain', 'Content-length: 900'));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $parameters);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
echo curl_exec($ch);
curl_close($ch);
}
It's pretty easy to not include headers in output. Just add option CURLOPT_HEADER and set it to false. But without headers how can you be sure that request is completed successfully?