I am using requests in python3 to send post request to yahoo login website, in Burpsuite I got the following response:
HTTP/1.0 200 OK
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Age: 0
Pragma: no-cache
Expires: 0
Referrer-Policy: origin-when-cross-origin
Cache-Control: no-cache, no-store, must-revalidate
set-cookie: AS=v=1&s=8ZJenFRo&d=A611b9f16|50....OXCxVzImJQI-~A; path=/; domain=login.yahoo.com; secure; HttpOnly
Content-Type: application/json; charset=utf-8
Content-Length: 149
Vary: Accept-Encoding
Date: Tue, 17 Aug 2021 11:20:50 GMT
Strict-Transport-Security: max-age=15552000
Server: ATS
{"location":"/account/challenge/recaptcha?done=https%3A%2F%2Fwww.yahoo.com%2F&sessionIndex=QQ--&acrumb=8ZJenFRo&display=login&authMechanism=primary"}
The goal is to get this line in python
{"location":"/account/challenge/recaptcha?done=https%3A%2F%2Fwww.yahoo.com%2F&sessionIndex=QQ--&acrumb=8ZJenFRo&display=login&authMechanism=primary"}
I wrote a simple script to do the post request but it crashes when response.json() is called
import requests
from user_agent import generate_user_agent
def check_yahoo(email):
yahoo_url = "https://login.yahoo.com"
data= {"username":f"{email}"}
heads = {"User-Agent":f"{generate_user_agent()}"}
response = requests.post(yahoo_url , params=data , headers=heads)
print (response.json())
return
#driver code
check_yahoo("some.one#yahoo.com")
How do you extract json data if it was broken
Error
File "/home/kali/Desktop/yah00/yah00.py", line 49, in <module>
check_yahoo(item)
File "/home/kali/Desktop/yah00/yah00.py", line 37, in check_yahoo
print (response.json())
File "/usr/lib/python3/dist-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
The json looks ok so it might be that the empty line is throwing it off. What is the content of response.text? If that includes the empty line, something like
json.loads(response.text.strip())
might be enough.
You can use a try/except to debug your error, like:
def some_function(data):
try:
*enter your success code*
except:
*enter in error statement*
see more information in https://docs.python.org/3/tutorial/errors.html
Related
I tried to deploy a few cloud functions. It just gets stuck.
I tried to deploy with gcloud cli and also using GUI.
Used command:
gcloud functions deploy hello --trigger-http --runtime=nodejs10 --verbosity=debug --log-http --region=us-central1
I tried to deploy to different regions. Same result.
logs:
DEBUG: Running [gcloud.functions.deploy] with arguments: [--log-http: "true", --region: "europe-west1", --runtime: "nodejs10", --trigger-http: "True", --verbosity: "debug", NAME: "hello"]
=======================
==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/expando-eve/locations/europe-west1/functions/hello?alt=json
method: GET
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 0
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/254.0.0 command/gcloud.functions.deploy invocation-id/fe5896cf229244f39b51b573c1477967 environment/None environment-version/None interactive/True from-script/False python/2.7.16 term/xterm-256color (Linux 5.0.0-21-generic)
== headers end ==
== body start ==
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,43,39"
cache-control: private
content-length: 158
content-type: application/json; charset=UTF-8
date: Thu, 25 Jul 2019 14:35:39 GMT
server: ESF
status: 404
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 0
-- headers end --
-- body start --
{
"error": {
"code": 404,
"message": "Function hello in region europe-west1 in project expando-eve does not exist",
"status": "NOT_FOUND"
}
}
-- body end --
total round trip time (request+response): 0.258 secs
---- response end ----
----------------------
INFO: Using ignore file at [./.gcloudignore].
DEBUG: Skipping file [./.gitignore]
DEBUG: Skipping file [./.gcloudignore]
DEBUG: Skipping file [./.idea/.gitignore]
INFO: Using ignore file at [./.gcloudignore].
DEBUG: Skipping file [.gitignore]
DEBUG: Skipping file [.gcloudignore]
DEBUG: Skipping file [.idea/.gitignore]
=======================
==== request start ====
uri: https://cloudfunctions.googleapis.com/v1/projects/expando-eve/locations/europe-west1/functions:generateUploadUrl?alt=json
method: POST
== headers start ==
Authorization: --- Token Redacted ---
accept: application/json
accept-encoding: gzip, deflate
content-length: 2
content-type: application/json
user-agent: google-cloud-sdk x_Tw5K8nnjoRAqULM9PFAC2b gcloud/254.0.0 command/gcloud.functions.deploy invocation-id/5d5a3c8af0f1441f99b9ad553e5cbbc2 environment/None environment-version/None interactive/True from-script/False python/2.7.16 term/xterm-256color (Linux 5.0.0-21-generic)
== headers end ==
== body start ==
{}
== body end ==
==== request end ====
---- response start ----
-- headers start --
-content-encoding: gzip
alt-svc: quic=":443"; ma=2592000; v="46,43,39"
cache-control: private
content-length: 122
content-type: application/json; charset=UTF-8
date: Thu, 25 Jul 2019 14:37:40 GMT
server: ESF
status: 503
transfer-encoding: chunked
vary: Origin, X-Origin, Referer
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 0
-- headers end --
-- body start --
{
"error": {
"code": 503,
"message": "The service is currently unavailable.",
"status": "UNAVAILABLE"
}
}
-- body end --
total round trip time (request+response): 120.153 secs
---- response end ----
----------------------
DEBUG: Response returned status 503, retrying
DEBUG: Retrying request to url https://cloudfunctions.googleapis.com/v1/projects/expando-eve/locations/europe-west1/functions:generateUploadUrl?alt=json after exception HttpError accessing <https://cloudfunctions.googleapis.com/v1/projects/expando-eve/locations/europe-west1/functions:generateUploadUrl?alt=json>: response: <{'status': '503', 'content-length': '122', 'x-xss-protection': '0', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Thu, 25 Jul 2019 14:37:40 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; ma=2592000; v="46,43,39"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
"error": {
"code": 503,
"message": "The service is currently unavailable.",
"status": "UNAVAILABLE"
}
}
I would expect that api would be available.
Yes, there was an issue from 25-07-2019 06:41 to 25-07-2019 18:23 PT, now it's solved you can check the status of Google Cloud Products at the Google Cloud Status Dashboard
I'm getting a response in JSON and I'm trying to use the
DeserializeStoreInfoResponse(Storeresponse As String) As String
Dim obj As New JSON_resultStorePos
obj = JsonConvert.DeserializeObject(Of JSON_resultStorePos)(Storeresponse)
the initial response is a HttpResponseMessage and all I need is the JSON string info. When it's a nice neat string in Json format everything deserializes fine into the JSON_resultStorePos table. But I'm trying to view the initial response to see if everything is there in their response and can't quite figure out where the actual JSON content is written in the HttpResponseMessage?
here's some of the response:
?aResponse
{StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
Strict-Transport-Security: max-age=7776000; includeSubDomains
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Pragma: no-cache
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Cache-Control: no-store
Date: Thu, 10 Nov 2016 13:37:39 GMT
Set-Cookie: SERVERID=04-97JSNQEU5RG2G; path=/; HttpOnly
Server: Apache
X-Powered-By: Express
Content-Length: 71
Content-Type: application/json
?aResponse.Content
{System.Net.Http.StreamContent}
Headers: {Content-Length: 71
Content-Type: application/json
}
IsBuffered: True
bufferSize: 4096
bufferedContent: {System.Net.Http.HttpContent.LimitMemoryStream}
canCalculateLength: True
content: {System.Net.Http.HttpClientHandler.WebExceptionWrapperStream}
contentConsumed: True
contentReadStream: Nothing
disposed: False
headers: {Content-Length: 71
Content-Type: application/json
How can I see the JSON string I'm deserializing ?
Background: I am using cfhttp and an auth token provided to me by a vendors API to do two GET calls. The first one fails the second is successful.
Call #1 is supposed to return a list of Order ID's in JSON format.
Call #2 uses one of those Order ID's to return a full order again in JSON format.
The calls are outlined below:
Request all of the ORDER ID's
<cfhttp method="get" url="https://test-9832.APIURL.com/criminal_api//1.0/service/requests" result="orderFirst">
<cfhttpparam type="Header" name="Authorization" value="Bearer 5BDCECFBDBE4680E74C8B81A58EAC544">
</cfhttp>
Request a full order based on the same auth token and one of the Order Ids returned in the previous step:
<cfhttp method="get" url="https://test-9832.APIURL.com/criminal_api//1.0/service/requests/WB-042916-YH59Z-PL-001" result="orderFull">
<cfhttpparam type="Header" name="Authorization" value="Bearer 5BDCECFBDBE4680E74C8B81A58EAC544">
</cfhttp>
The calls are using the same exact cfhttpparam and value the only difference is the URL. The vendor mentioned they sometimes see issues if the Auth Token is double encoded. I am setting the auth token value statically at this point just to get it to work.
Below is a cfdump of what is returned in Step #1 that fails:
struct
Charset UTF-8
ErrorDetail [empty string]
Filecontent Connection Failure
Header HTTP/1.1 200 OK Connection: close Expires: Wed, 31 Dec 1969 16:00:00 PST Date: Wed, 14 Sep 2016 04:47:01 GMT Server: hws Pragma: No-cache Cache-Control: no-cache Set-Cookie: X-HR-ClientSessionId=2_12.161.115.226_1473828421271;Secure; path=/; HttpOnly Content-Type: application/json;charset=UTF-8
Mimetype application/json
Responseheader
struct
Cache-Control no-cache
Connection close
Content-Type application/json;charset=UTF-8
Date Wed, 14 Sep 2016 04:47:01 GMT
Expires Wed, 31 Dec 1969 16:00:00 PST
Explanation OK
Http_Version HTTP/1.1
Pragma No-cache
Server hws
Set-Cookie X-HR-ClientSessionId=2_12.161.115.226_1473828421271;Secure; path=/; HttpOnly
Status_Code 200
Statuscode 200 OK
Text NO
The status code is showing 200 OK but the file contents is Connection Failure.
QUESTION: Is there anything else I can add to my script to help debug or test?
I was calling a Rest API locally from terminal. This was like below:
http -v -f --timeout=60 GET 'http://localhost:8080/api/v1/public/users/signin?email=myemail#email.com&password=mypassword'
It is returning following output:
GET /api/v1/public/users/signin?email=myemail#email.com&password=mypassword HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:8080
User-Agent: HTTPie/0.8.0
HTTP/1.1 400 Bad Request
Content-Type: application/json
Date: Mon, 28 Mar 2016 16:59:08 GMT
Set-Cookie: rememberMe=deleteMe; Path=/; Max-Age=0; Expires=Sun, 27- Mar-2016 16:59:13 GMT
Transfer-Encoding: chunked
{
"debugMessage": "Submitted credentials for token [org.apache.shiro.authc.UsernamePasswordToken - najmul#qianalysis.com, rememberMe=true] did not match the expected credentials.Submitted credentials for token [org.apache.shiro.authc.UsernamePasswordToken - najmul#qianalysis.com, rememberMe=true] did not match the expected credentials.: The subject was expected to be true, but was false",
"errorCode": "INCORRECT_CREDENTIALS",
"logId": "50a695c1a81e0542",
"stackTrace": "org.niopack.r.....
}
Now I want to read this JSON data using Alamofire. I tried with responseJSON serialiser with following way:
Alamofire.request(method, urlString, parameters: parameters, encoding: .URL, headers: cookieHeader)
.validate()
.responseJSON {
alamofireResponse in
let jsonData = alamofireResponse.result.value
}
But this I am getting jsonData as nil. What you will do if you were are here?
Thanks.
For me removing the validate() allowed me to see the reason the request was failing. For some reason when validate fails, the data of the response is not parsed.
I'm using google api dot net client, and i'm a bit confused.
Following code returns me the entire list of a specific folder (with an Id: XXXXXXXXXXXXXXXXXXXXX).
List<File> result = new List<File>();
FilesResource.ListRequest request = service.Files.List();
request.Q = "'XXXXXXXXXXXXXXXXXXXXX' in parents";
request.MaxResults = 35;
FileList files = request.Fetch();
As example one item of this FileList is
"id": "0B5XhOfl0NZ2cZ1M1aVBPcVlTUTA",
"title": "87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC"
If i change the query, searching for a title, i receive always an empty FileList.
List<File> result = new List<File>();
FilesResource.ListRequest request = service.Files.List();
request.Q = "title = '87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC'";
request.MaxResults = 35;
FileList files = request.Fetch();
It does'nt matter, if the operator is '=' or 'contains', or by searching only a part of the directory name:
title contains '87BC1CAE'
I receive correct results only by searching for IDs.
Following code works flawless:
File file = service.Files.Get(fileId).Fetch();
Console.WriteLine("Title: " + file.Title);
Console.WriteLine("Description: " + file.Description);
Console.WriteLine("MIME type: " + file.MimeType);
Help please :)
#AliAfshar
Raw HTTP responses:
1st case (title = '87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC')
request.Fetch()
TestGoogleApi.vshost.exe Information: 0 : DotNetOpenAuth, Version=4.0.0.11165, Culture=neutral, PublicKeyToken=2780ccd10d57b246 (official)
TestGoogleApi.vshost.exe Information: 0 : Preparing to send AssertionFlowMessage (2.0) message.
TestGoogleApi.vshost.exe Information: 0 : Sending AssertionFlowMessage request.
TestGoogleApi.vshost.exe Information: 0 : HTTP POST https://accounts.google.com/o/oauth2/token
TestGoogleApi.vshost.exe Information: 0 : The following required parameters were missing from the DotNetOpenAuth.OAuth2.Messages.AccessTokenFailedResponse message: {error,}
TestGoogleApi.vshost.exe Information: 0 : Received UnauthorizedResponse response.
{Google.Apis.Drive.v2.Data.FileList}
_etag: "\"Q0cVodxX8sh4vfxZTlOyWcmmc0k/vyGp6PvFo4RvsFtPoIWeCReyIC8\""
_items: Count = 0
_kind: null
_nextLink: null
_nextPageToken: null
_selfLink: null
ETag: "\"Q0cVodxX8sh4vfxZTlOyWcmmc0k/vyGp6PvFo4RvsFtPoIWeCReyIC8\""
Items: Count = 0
Kind: null
NextLink: null
NextPageToken: null
SelfLink: null
2nd case ('XXXXXXXXXXXXXXXXXXXXXX' in parents )
request.Fetch()
TestGoogleApi.vshost.exe Information: 0 : DotNetOpenAuth, Version=4.0.0.11165, Culture=neutral, PublicKeyToken=2780ccd10d57b246 (official)
TestGoogleApi.vshost.exe Information: 0 : Preparing to send AssertionFlowMessage (2.0) message.
TestGoogleApi.vshost.exe Information: 0 : Sending AssertionFlowMessage request.
TestGoogleApi.vshost.exe Information: 0 : HTTP POST https://accounts.google.com/o/oauth2/token
TestGoogleApi.vshost.exe Information: 0 : The following required parameters were missing from the DotNetOpenAuth.OAuth2.Messages.AccessTokenFailedResponse message: {error,}
TestGoogleApi.vshost.exe Information: 0 : Received UnauthorizedResponse response.
{Google.Apis.Drive.v2.Data.FileList}
_etag: "\"Q0cVodxX8sh4vfxZTlOyWcmmc0k/Jyx7utsp71-_JwU5RHnI_VZmL5o\""
_items: Count = 1575
_kind: null
_nextLink: null
_nextPageToken: null
_selfLink: null
ETag: "\"Q0cVodxX8sh4vfxZTlOyWcmmc0k/Jyx7utsp71-_JwU5RHnI_VZmL5o\""
Items: Count = 1575
Kind: null
NextLink: null
NextPageToken: null
SelfLink: null
Updated on 2013 Feb 02
Ok excuse me for delay.
i used fiddler and here is captured result:
first request:
POST https://accounts.google.com/o/oauth2/token HTTP/1.1
Content-Type: application/x-www-form-urlencoded; charset=utf-8
User-Agent: DotNetOpenAuth/4.0.0.11165
Host: accounts.google.com
Cache-Control: no-store,no-cache
Pragma: no-cache
Content-Length: 603
Connection: Keep-Alive
grant_type=assertion&assertion_type=http%3A%2F%2Foauth.net%2Fgrant_type%2Fjwt%2F1.0%2Fbearer&assertion=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiI0MjYwNjE5OTE1NzktYXEzMHBhcmVsYmpsb3BrMjlqcXFvdjhsdWhic2o3YjdAZGV2ZWxvcGVyLmdzZXJ2aWNlYWNjb3VudC5jb20iLCJzY29wZSI6Imh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvZHJpdmUiLCJhdWQiOiJodHRwczovL2FjY291bnRzLmdvb2dsZS5jb20vby9vYXV0aDIvdG9rZW4iLCJleHAiOjEzNjAwNjA0NDMsImlhdCI6MTM2MDA1Njg0M30.dk7Vdu-................-LIw0sFrVko-VWL7-elhz59VQcU_.........
first answer:
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Date: Tue, 05 Feb 2013 09:34:03 GMT
Content-Type: application/json
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Server: GSE
Content-Length: 128
{
"access_token" : "ya29.AHES6ZS-...........-a8HRHfMZ-NxXekx",
"token_type" : "Bearer",
"expires_in" : 3600
}
second request:
GET https://www.googleapis.com/drive/v2/files?alt=json&fields=items(id%2Ctitle)%2CnextPageToken&prettyPrint=true&maxResults=2000&q=title%20%3D%20'87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC' HTTP/1.1
Authorization: Bearer ya29.AHES6ZS-...........-a8HRHfMZ-NxXekx
Content-Type: application/json; charset=utf-8
User-Agent: TestGoogleApi google-api-dotnet-client/ Win32NT/6.1.7600.0 (gzip)
Host: www.googleapis.com
Accept-Encoding: gzip, deflate
second answer:
HTTP/1.1 200 OK
Expires: Tue, 05 Feb 2013 09:34:04 GMT
Date: Tue, 05 Feb 2013 09:34:04 GMT
Cache-Control: private, max-age=0, must-revalidate, no-transform
ETag: "Q0cVodxX8sh4vfxZTlOyWcmmc0k/vyGp6PvFo4RvsFtPoIWeCReyIC8"
Content-Type: application/json; charset=UTF-8
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Content-Length: 17
Server: GSE
{
"items": []
}
as you see items count is 0.
If i make the same request via web interface
https://developers.google.com/drive/v2/reference/files/list it gives me 2 items, as it should be.
This is made with https://developers.google.com/oauthplayground/
GET /drive/v2/files?alt=json&fields=items(id%2Ctitle)%2CnextPageToken&prettyPrint=true&maxResults=2000&q=title%20%3D%20'87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC' HTTP/1.1
Host: www.googleapis.com
Content-length: 0
Authorization: OAuth ya29.AHES6ZSDr7bDFMQxjR........ObzyEhkzQmN
HTTP/1.1 200 OK
Content-length: 220
Via: HTTP/1.1 GWA
Content-location: https://www.googleapis.com/drive/v2/files?alt=json&fields=items(id%2Ctitle)%2CnextPageToken&prettyPrint=true&maxResults=2000&q=title%20%3D%20'87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC'
X-content-type-options: nosniff
Etag: "Q0cVodxX8sh4vfxZTlOyWcmmc0k/oUg90Ml6_g3EsaXCogiJOEMkZ-M"
X-google-cache-control: remote-fetch
-content-encoding: gzip
Server: GSE
Reason: OK
X-xss-protection: 1; mode=block
Cache-control: private, max-age=0, must-revalidate, no-transform
Date: Tue, 05 Feb 2013 15:30:09 GMT
X-frame-options: SAMEORIGIN
Content-type: application/json; charset=UTF-8
Expires: Tue, 05 Feb 2013 15:30:09 GMT
{
"items": [
{
"id": "0B5XhOf....VlTUTA",
"title": "87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC"
},
{
"id": "0B5XhO....EbGRLVVVReGM",
"title": "87BC1CAE-A01C-43A1-BB21-BE3DA6D6C7DC"
}
]
}
Thank you very much, i really appreciate your help
Looks like the confusion is due to the fact that you are using a service account in your code, but comparing the results with what you see in the web interface.
When using the web interface or the OAuth Playground, you are authorizing the app to use your personal account and not a service account. The service account doesn't have access to the same files you have access to, as it is actually an application-owned account.
If you want to use a service account to access your files, you should perform domain-wide delegation as explained at https://developers.google.com/drive/delegation
This still doesn't explain why some queries return results and others don't, but please try everything again without using service accounts and update the issue if things still don't work as expected.