For learning purposes I'm trying to reproduce Instagram internal API with Ruby and Faraday. However, the response's body I get when making a POST is somehow encoded instead of JSON:
What the response's body should look like:
{
"status": "ok",
"media": {
"page_info": {
"start_cursor": "1447303180937779444_4460593680",
"has_next_page": true,
"end_cursor": "1447303180937779444",
"has_previous_page": true
},
...
What I get:
#=> \x1F\x8B\b\x00#\x15\x9EX\x02\xFF...
Question:
Any idea (i) why I'm getting a response's body like that and (ii) how can I convert that to JSON?
Flow:
When you hit https://www.instagram.com/explore/locations/127963847/madrid-spain/ in your browser Instagram makes two requests (among others):
GET: https://www.instagram.com/explore/locations/127963847/madrid-spain/
POST: https://www.instagram.com/query/
I used Postman to intercept requests and just copied headers and parameters for the second (/query/) request. This is my implementation (get status '200'):
class IcTest
require 'open-uri'
require "net/http"
require "uri"
def self.faraday
conn = Faraday.new(:url => 'https://www.instagram.com') do |faraday|
faraday.request :url_encoded # form-encode POST params
faraday.response :logger # log requests to STDOUT
faraday.adapter Faraday.default_adapter # make requests with Net::HTTP
end
res = conn.post do |req|
req.url '/query/'
req.headers['Origin'] = 'https://www.instagram.com'
req.headers['X-Instagram-AJAX'] = '1'
req.headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36'
req.headers['Content-Type'] = 'application/x-www-form-urlencoded'
# req.headers['Accept'] = '*/*'
req.headers['X-Requested-With'] = 'XMLHttpRequest'
req.headers['X-CSRFToken'] = 'SrxvROytxQHAesy1XcgcM2PWrEHHuQnD'
req.headers['Referer'] = 'https://www.instagram.com/explore/locations/127963847/madrid-spain/'
req.headers['Accept-Encoding'] = 'gzip, deflate, br'
req.headers['Accept-Language'] = 'es,en;q=0.8,gl;q=0.6,pt;q=0.4,pl;q=0.2'
req.headers['Cookie'] = 'mid=SJt50gAEAAE6KZ50GByVoStJKLUH; sessionid=IGSC514a2e9015f548b09176228f83ad5fe716f32e7143f6fe710c19a71c08b9828b%3Apc2KPxgwvokLyZhfZHcO1Qzfb2mpykG8%3A%7B%22_token%22%3A%2233263701%3Ai7HSIbxIMLj70AoUrCRjd0o1g7egHg79%3Acde5fe679ed6d86011d70b7291901998b8aae7d0aaaccdf02a2c5abeeaeb5908%22%2C%22asns%22%3A%7B%2283.34.38.249%22%3A3352%2C%22time%22%3A1486584547%7D%2C%22last_refreshed%22%3A1436584547.2838287%2C%22_platform%22%3A4%2C%22_token_ver%22%3A2%2C%22_auth_user_backend%22%3A%22accounts.backends.CaseInsensitiveModelBackend%22%2C%22_auth_user_id%22%3A33233701%2C%22_auth_user_hash%22%3A%22%22%7D; ds_user_id=31263701; csrftoken=sxvROytxQHAesy1XcgcM2PWrEHHuQnD; s_network=""; ig_vw=1440; ig_pr=2;'
req.body = { :q => "ig_location(127963847) { media.after('', 60) { count, nodes { caption, code, comments { count }, comments_disabled, date, dimensions { height, width }, display_src, id, is_video, likes { count }, owner { id }, thumbnail_src, video_views }, page_info} }",
:ref => "locations::show",
:query_id => "" }
end
end
Thanks.
Josh comment made it! :-)
The body's content was gzip.
Solution here.
Related
Old Question on Stack Overflow< https://stackoverflow.com/questions/61073325/vba-xml-web-log-in-to-usga-ghin-website-not-working>
USGA Webiste https://www.ghin.com/login
Page 2 - USGA https://www.ghin.com/golfer-lookup/following
I built an Excel VBA app that uses the data collected from and API/JSON data-pull from the USGA website, of which I am an authorized USER with a valid account and password. However, the code which I have used reliably for about 2 years is now generating a "Invalid Token Error".
The "Invalid Token Error" may be password related. My prior code required no password input. I have tired to build the password input into the input/response but as of yet no luck?
Any thoughts on you to solve "Invalid Token Error" and possibly, construct the password input on my part? Here is may old code (Also posted on the Stack Overflow links above)
Sub GetInformation()
Const Url = "https://api2.ghin.com/api/v1/public/login.json?"
Dim Http As New XMLHTTP60, ghinNum$, lastName$
ghinNum = "" 'put your ghinNum here
lastName = "" 'put your lastName here
With Http
.Open "GET", Url & "ghinNumber=" & ghinNum & "&lastName=" & lastName & "&remember_me=false", False
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36"
.setRequestHeader "Referer", "https://www.ghin.com/login"
.send
End With
MsgBox Http.responseText
End Sub
Same here. My page is in PHP, so you will have to adjust accordingly to VBA. It looks like they are now requiring the password whereas it used to work with just the last name and ghin number. I use POST below with the array I show in the $postdata below. But GET may still work. Good luck.
<?PHP
// my new PHP login code
$ghin = $my_info['ghin']; // 7-digit number
$pw = $my_info['ghinpw'];
$postdata = json_encode(
array(
'user' => array(
'email_or_ghin' => $ghin,
'password' => $pw,
'remember_me' => 'true',
),
'token' => 'nonblank'
)
);
// gave me warning about wanting a nonblank token, so I gave it one!
$options = array('http' =>
array(
'header' =>
"Content-type: application/json\n" .
"Accept: application/json\n" .
"user_agent: Mozilla/5.0 (X11; CrOS x86_64 14469.16.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.100.4844.23 Safari/537.36",
'method' => 'POST',
'content' => $postdata
)
);
$context = stream_context_create($options);
$url = "https://api2.ghin.com/api/v1/golfer_login.json";
$contents = file_get_contents($url, false, $context);
$results = json_decode($contents, true);
$token = $results['golfer_user']['golfer_user_token'];
$options = array('http' => array(
'method' => 'GET',
'header' => 'Authorization: Bearer ' . $token
));
$context = stream_context_create($options);
// then you can grab URLs
$url = "https://api.ghin.com/api/v1/golfers/$ghin/handicap_history_count.json?rev_count=3";
$contents = file_get_contents($url, false, $context);
if ($contents) {
$results = json_decode($contents, true);
$handi = $results['handicap_revisions'][0]['Value'];
}
?>
I too have an app that used the GHIN API as well. It appears they simply removed their public endpoint (https://api2.ghin.com/api/v1/public). You now have to authenticate to get a token to use their API.
Here's a site with some really good documentation for the GHIN API: https://app.swaggerhub.com/apis-docs/GHIN/Admin/1.0
Sorry that's not the answer you wanted to hear, I'm sure.
EDIT: Now I have a GHIN and can log in. Here's my working Python code:
import requests
import json
ghinNum = "GHINID"
password = "GHINPASS"
headers = {
"Content-Type": "application/json; charset=utf-8",
"Accept": "application/json",
}
data = {
"user": {
"email_or_ghin": ghinNum,
"password": password,
"remember_me": "true",
},
"token": "nonblank",
}
r = requests.post("https://api2.ghin.com/api/v1/golfer_login.json", headers=headers, json=data)
headers["Authorization"] = "Bearer " + r.json()["golfer_user"]["golfer_user_token"]
url = "https://api.ghin.com/api/v1/golfers/search.json?per_page=1&page=1&golfer_id=" + ghinNum
r = requests.get(url, headers=headers)
if r.status_code == 200:
data = r.json()
handicap_index = data['golfers'][0]['handicap_index']
print(handicap_index)
else:
print(r.status_code)
Guys In you have an Active GHIN then it's easy.
Download TheGrint.
Register.
Link your GHIN to your newly created account.
Start posting your scores from TheGrint to the USGA
Get you handicap revise overnight.
As simple as that.
Enjoy TheGrint.
I'm an absolute beginner in get/post requests and micropython.
I'm programming my ESP8266 Wemos D1 mini as a HTTP server with micropython. My project consists of using a website to control the RGB values of a neopixel matrix hooked up to the D1 (all the code is on my GitHub here: https://github.com/julien123123/NeoLamp-Micro).
Basically, the website contains three sliders: one for Red, one for Green and one for Blue. A javascript code reads the value of each slider and sends it to the micropython code with using the POST method as follows :
getColors = function() {
var rgb = new Array(slider1.value, slider2.value, slider3.value);
return rgb;
};
postColors = function(rgb) {
var xmlhttp = new XMLHttpRequest();
var npxJSON = '{"R":' + rgb[0] + ', "G":' + rgb[1] + ', "B":' + rgb[2] + '}';
xmlhttp.open('POST', 'http://' + window.location.hostname + '/npx', true);
xmlhttp.setRequestHeader('Content-type', 'application/json');
xmlhttp.send(npxJSON);
};
To recieve the resquest in micropython here's my code:
conn, addr = s.accept()
request = conn.recv(1024)
request = str(request)
print(request)
The response prints as follows:
b'POST /npx HTTP/1.1\r\nHost: 192.xxx.xxx.xxx\r\nConnection: keep-alive\r\nContent-Length: 27\r\nOrigin: http://192.168.0.110\r\nUser-Agent: Mozilla/5.0 (X11; CrOS x86_64 10323.46.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.107 Safari/537.36\r\nContent-type: application/json\r\nAccept: */*\r\nReferer: http://192.xxx.xxx.xxx/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: fr,en;q=0.9,fr-CA;q=0.8\r\n\r\n{"R":114, "G":120, "B":236}'
The only important bit for me is at the end : {"R":114, "G":120, "B":236}. I want to use those values to change the color values of my neopixel object.
My question to you is how to I process the response so that I keep only the dictionary containing the RGB variables at the end of the response??
Thanks in advance (I'm almost there!)
This is more related to generic python data type. The data type of request is in bytes as indicated by prefix b in b'POST /npx HTTP/1.1...\r\n{"R":114, "G":120, "B":236}'. You will have to use decode() to convert it to string
import json
request = b'POST /npx HTTP/1.1\r\nHost: 192.xxx.xxx.xxx\r\nConnection: keep-alive\r\nContent-Length: 27\r\nOrigin: http://192.168.0.110\r\nUser-Agent: Mozilla/5.0 (X11; CrOS x86_64 10323.46.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.107 Safari/537.36\r\nContent-type: application/json\r\nAccept: */*\r\nReferer: http://192.xxx.xxx.xxx/\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: fr,en;q=0.9,fr-CA;q=0.8\r\n\r\n{"R":114, "G":120, "B":236}'
data = request.decode() # convert to str
rgb = data.split('\r\n')[-1:] #split the str and discard the http header
for color in rgb:
print(color, type(color))
d = json.loads(color)
print(d, type(d))
The result of color is a str representation of an json object, the d will give you a python dict object to be used for further manipulation:
{"R":114, "G":120, "B":236} <class 'str'>
{'R': 114, 'G': 120, 'B': 236} <class 'dict'>
I use the following filters configuration:
filter {
if [type] == "client-log" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
urldecode{
field => "request"
}
mutate {
gsub => [ "request", '/log/', '' ]
}
json {
source => "request"
}
}
}
It works fine when request is only a one level json object. If it gets more then one level Logstash is getting parse error. Any advice?
request example:
{
session_id: "123",
message: {
id: 1221
//....
}
//....
}
error:
{:timestamp=>"2016-09-07T15:53:34.712000+0100", :message=>"Error parsing json", :source=>"request", :raw=>"{\"session_id\":\"8d078da0-74f9-11e6-8d31-6925e76dde0e\",\"level\":\"Debug\",\"methodName\":\"fetchExternalData\",\"class\":\"fetchExternalData\",\"lineNumber\":78,\"message\":\"{\"0\":\"fetchExternalData doFetch assets \",\"1\":\"http://lab6services:8080/alerts\",\"2\":{\"method\":\"post\",\"headers\":{},\"body\":\"{\\\"results\\\":{\\\"types\\\":[\\\"alerts\\\"],\\\"format\\\":\\\"table\\\"},\\\"filter\\\":{\\\"title\\\":\\\"new alerts\\\",\\\"filterType\\\":\\\"PROPERTY\\\",\\\"operator\\\":\\\"range\\\",\\\"field\\\":\\\"createdAt\\\",\\\"type\\\":\\\"alert\\\",\\\"values\\\":[{\\\"value\\\":\\\"07/09/2016 15:49:44\\\"},{\\\"value\\\":\\\"07/09/2016 15:51:44\\\"}]},\\\"aggregate\\\":null}\"}}\",\"version\":\"1.2.0\",\"user\":\"user\",\"timestamp\":\"2016-09-07T12:51:44.953Z\"}", :exception=>#<LogStash::Json::ParserError: Unexpected character ('0' (code 48)): was expecting comma to separate OBJECT entries
at [Source: [B#51b925ff; line: 1, column: 161]>, :level=>:warn}
log line:
127.8.4.1 - - [07/Sep/2016:15:54:07 +0100] "GET /log/%7B%22session_id%22:%228d078da0-74f9-11e6-8d31-6925e76dde0e%22,%22level%22:%22Debug%22,%22methodName%22:%22fetchExternalData%22,%22class%22:%22fetchExternalData%22,%22lineNumber%22:78,%22message%22:%22%7B%220%22:%22fetchExternalData%20doFetch%20assets%20%22,%221%22:%22http://lab6services:8080/alerts%22,%222%22:%7B%22method%22:%22post%22,%22headers%22:%7B%7D,%22body%22:%22%7B%5C%22results%5C%22:%7B%5C%22types%5C%22:%5B%5C%22alerts%5C%22%5D,%5C%22format%5C%22:%5C%22table%5C%22%7D,%5C%22filter%5C%22:%7B%5C%22title%5C%22:%5C%22new%20alerts%5C%22,%5C%22filterType%5C%22:%5C%22PROPERTY%5C%22,%5C%22operator%5C%22:%5C%22range%5C%22,%5C%22field%5C%22:%5C%22createdAt%5C%22,%5C%22type%5C%22:%5C%22alert%5C%22,%5C%22values%5C%22:%5B%7B%5C%22value%5C%22:%5C%2207/09/2016%2015:49:44%5C%22%7D,%7B%5C%22value%5C%22:%5C%2207/09/2016%2015:52:22%5C%22%7D%5D%7D,%5C%22aggregate%5C%22:null%7D%22%7D%7D%22,%22version%22:%221.2.0%22,%22user%22:%22user%22,%22timestamp%22:%222016-09-07T12:52:22.928Z%22%7D HTTP/1.1" 200 0 "http://localhost:3000/main.worker.js" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"
--- EDIT ---
I want to parse only the first level of 'resquest'. How can I prevent the filter to parse any nested json elements?
I am starting development with Erlang and need to make a REST HTTP call to a server where I send a JSON and receive JSON confirmation.
Follows the code
Method = put,
URL = "http://api.teste.com:8080/v1/user_auth",
Header = [],
Type = "application/json",
Json = <<"{ \"data\" : { \"test-one\" : \"123\", \"test-two\" : \"return test 2\" } }">>,
HTTPOptions = [],
Options = [],
application:start(ssl),
application:start(inets),
httpc:request(Method, {URL, Header, Type, Json}, HTTPOptions, Options).\
When you run this code I am with the following error:
=ERROR REPORT==== 5-Dec-2015::14:21:01 ===
Error in process <0.161.0> on node 'middleware#127.0.0.1' with exit value:
{[{reason,undef},
{mfa,{user_account_handler,handle_post,2}},
{stacktrace,[{httpc,request,
[put,
{"http://api.teste.com:8080/v1/user_auth",[],
"application/json",
<<"{ \"data\" : { \"test-one\" : \"123\", \"test-two\" : \"return test 2\" } }">>},
[],[]],
[]},
{cowboy_rest,call,3,[{file,"src/cowboy_rest.erl"},{line,972}]},
{cowboy_rest,process_content_type,3,
[{file,"src/cowboy_rest.erl"},{line,773}]},
{cowboy_protocol,execute,4,
[{file,"src/cowboy_protocol.erl"},
{line,442}]}]},
{req,[{socket,#Port<0.479>},
{transport,ranch_tcp},
{connection,keepalive},
{pid,<0.161.0>},
{method,<<"POST">>},
{version,'HTTP/1.1'},
{peer,{{127,0,0,1},49895}},
{host,<<"localhost">>},
{host_info,undefined},
{port,8080},
{path,<<"/v1/create_user_account">>},
{path_info,undefined},
{qs,<<>>},
{qs_vals,undefined},
{bindings,[]},
{headers,[{<<"host">>,<<"localhost:8080">>},
{<<"connection">>,<<"keep-alive">>},
{<<"content-length">>,<<"58">>},
{<<"cache-control">>,<<"no-cache">>},
{<<"origin">>,
<<"chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop">>},
{<<"content-type">>,<<"application/json">>},
{<<"user-agent">>,
<<"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36">>},
{<<"postman-token">>,
<<"2dc302f2-7c93-b9f9-2143-cff41bfeb45a">>},
{<<"accept">>,<<"*/*">>},
{<<"accept-encoding">>,<<"gzip, deflate">>},
{<<"accept-language">>,
<<"en-US,en;q=0.8,es;q=0.6,pt;q=0.4">>}]},
{p_headers,[{<<"content-type">>,{<<"application">>,<<"json">>,[]}},
{<<"if-modified-since">>,undefined},
{<<"if-none-match">>,undefined},
{<<"if-unmodified-since">>,undefined},
{<<"if-match">>,undefined},
{<<"accept">>,[{{<<"*">>,<<"*">>,[]},1000,[]}]},
{<<"connection">>,[<<"keep-alive">>]}]},
{cookies,undefined},
{meta,[{media_type,{<<"application">>,<<"json">>,[]}},
{charset,undefined}]},
{body_state,waiting},
{buffer,<<"{\n \"username\":\"igor#gmail.com\"\n , \"password\":\"123\"\n}">>},
{multipart,undefined},
{resp_compress,false},
{resp_state,waiting},
{resp_headers,[{<<"content-type">>,
[<<"application">>,<<"/">>,<<"json">>,<<>>]}]},
{resp_body,<<>>},
{onresponse,undefined}]},
{state,undefined}],
[{cowboy_rest,process_content_type,3,
[{file,"src/cowboy_rest.erl"},{line,773}]},
{cowboy_protocol,execute,4,[{file,"src/cowboy_protocol.erl"},{line,442}]}]}
Sample code:
sub record_put :Private {
my ( $self, $c, #args ) = #_;
$c->log->info( join ', ', %{ $c->request->headers } ) ;
$c->log->info( $c->request->body ) ;
$c->response->body( $c->request->body ) ;
}
Here's the log data:
[info] user-agent, Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/28.0.1500.71 Chrome/28.0.1500.71 Safari/537.36, connection, keep-alive, accept, application/json, text/javascript, */*; q=0.01, accept-language, en-US,en;q=0.8, x-requested-with, XMLHttpRequest, origin, http://localhost:3000, accept-encoding, gzip,deflate,sdch, content-length, 125, host, localhost:3000, ::std_case, HASH(0xaec0ba0), content-type, application/json, referer, http://localhost:3000/test
[info] /tmp/PM2C6FXpcC
Here's a snippet of text from the Catalyst::Request document:
$req->body
Returns the message body of the request, as returned by HTTP::Body: a string, unless Content-Type is application/x-www-form-urlencoded, text/xml, or multipart/form-data, in which case a File::Temp object is returned.
The File::Temp manpage does not help. Even the 'object' overloads its stringification, I can't see how to extract the contents.
Here's what I used:
my $rbody = $c->req->body;
if ($rbody) {
# Post requests are stored on the filesystem under certain obscure conditions,
# in which case $rbody is a filehandle pointing to the temporary file
if (ref $rbody) { # a filehandle
$content = join "", readline($rbody);
close $rbody;
unlink "$rbody"; # filehandle stringifies to name of temp file
} else { # a string
$content = $rbody;
}
}
The thing you get back from the body method represents a temporary file, and can be treated like a filehandle or like a string. if you treat it like a filehandle, it reads from the temporary file; if used like a string, its value is the name of the temporary file. I used the seldom-seen builtin function readline, which is the same as the more common <…> operator.
I don't expect the else path to ever be taken, but it's there defensively, because you never know.
Added 2014-06-09: You need the explicit close; otherwise the code has a file descriptor leak. Catalyst devs claim that it should be cleaning up the handle automatically, but it doesn't.
if you are just trying to parse JSON, the newest stable Catalyst has a method 'body_data' that does this for you (see: http://www.catalystframework.org/calendar/2013/6)