I want to execute this request with cookies that I have saved in file(cookies are good because i can execute other requests to page)
http://www.banggood.com/index.php?com=event&t=recordSignInShare&fb_id=197203087314503_251208398580638&code=
Yes, last parameter should be empty.
When I open it in browser it gives me json data (no matter how they look)
but when i try to do the same request on vb.net app it redirects me to another page.
Here is code from VB:
Dim postData As String = "com=event&t=recordSignInShare&fb_id=197203087314503_251208398580638&code="
Dim bytes() As Byte = ASCIIEncoding.UTF8.GetBytes(postData)
Dim postReq As HttpWebRequest = WebRequest.Create("http://www.banggood.com/index.php")
postReq.Method = "POST"
postReq.KeepAlive = True
postReq.CookieContainer = cookies
'postReq.ContentType = "application/x-www-form-urlencoded; charset=UTF-8"
postReq.Referer = "http://www.banggood.com/"
postReq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"
postReq.ContentLength = bytes.Length
Dim postStream As Stream = postReq.GetRequestStream()
postStream.Write(bytes, 0, bytes.Length)
postStream.Close()
Dim postResponse As HttpWebResponse
postResponse = postReq.GetResponse()
cookies.Add(postResponse.Cookies)
Dim reader As New StreamReader(postResponse.GetResponseStream())
Dim strSource As String = reader.ReadToEnd
Return strSource
And it returns me html code, not json data :(
This is how looks request from network monitor when i open it by browser
Request:
Accept:application/json, text/javascript, */*; q=0.01
Accept-Encoding:gzip, deflate, sdch
Accept-Language:pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Host:www.banggood.com
Referer:http://www.banggood.com/2016midyear.html?utmid=796
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36
X-Requested-With:XMLHttpRequest
Response:
Cache-Control:max-age=0, no-cache, no-store
Connection:keep-alive
Content-Length:94
Content-Type:text/html; charset=utf-8
I think that original(from code) request/response headers aren't important because I can execute it proper just by opening this link in browser.
Problem was, that i need get request :P
Thanks the_lotus
Dim urlphp As String = "" & dominio & "" & carpetanoti & "/demanda_alta.php"
Dim Conexion As HttpWebRequest = CType(WebRequest.Create(urlphp), HttpWebRequest)
Conexion.Method = "POST"
Conexion.ContentType = "application/x-www-form-urlencoded"
Dim POST_DATA As String = ("&cTitulo=" & "Te necesitamos!!" & "&cMensaje=" & "ALTA DEMANDA TENEMOS DOMICILIOS PARA TI" & "")
Dim byteArray() As Byte = Encoding.UTF8.GetBytes(POST_DATA)
Conexion.ContentLength = byteArray.Length
Dim FLUJO As Stream = Conexion.GetRequestStream()
FLUJO.Write(byteArray, 0, byteArray.Length)
FLUJO.Close()
Dim Response As HttpWebResponse = Conexion.GetResponse()
FLUJO = Response.GetResponseStream()
Dim LEER As New StreamReader(FLUJO)
Dim ServerResponse As String = LEER.ReadToEnd()
LEER.Close()
FLUJO.Close()
Response.Close()
Related
i used chromedriver but its slowly,i tried webbrowser but not worked.
My webpage is;
**https://coinmarketcap.com/converter/btc/usd/?amt=1 **
so
i want get of how much usd one bitcoin or more then one
Coinmarketcap uses it's own API service. You can use their API instead of using web page. Dont forget to add Newtonsoft library to your project from Nuget.
Dim btcAmount = 4
Dim tempcookies As New CookieContainer
Dim encoding As New UTF8Encoding
Dim postreq As HttpWebRequest = DirectCast(HttpWebRequest.Create("https://api.coinmarketcap.com/data-api/v3/tools/price-conversion?amount=" & btcAmount & "&convert_id=2781&id=1"), HttpWebRequest)
postreq.Method = "GET"
postreq.KeepAlive = True
postreq.CookieContainer = tempcookies
postreq.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR 3.5.30729"
postreq.ContentType = "application/x-www-form-urlencoded"
postreq.Referer = "https://coinmarketcap.com"
Dim postresponse As HttpWebResponse
postresponse = DirectCast(postreq.GetResponse, HttpWebResponse)
tempcookies.Add(postresponse.Cookies)
Dim postreqreader As New StreamReader(postresponse.GetResponseStream())
Dim result As String = postreqreader.ReadToEnd
Dim json As JObject = JObject.Parse(result)
MessageBox.Show(json.Item("data").Item("quote")(0).Item("price"))
I'm trying to parse property information from this link which produces a JSON response. I've used here JSON and VBA converter. However, when I run the script below, I get an error keyNotFoundError. I'm trying to parse the value of properties which is within features.
Public Sub parseJson()
Dim jsonObject As Object, oElem As Variant
Dim resp$, Url$, R&
Url = "https://torontolife.com/wp-content/themes/sjm-underscores/inc/neighbourhoods/2015/compiled.json"
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", Url, False
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36"
.send
resp = .responseText
End With
Set jsonObject = JsonConverter.parseJson(resp)
For Each oElem In jsonObject("features")
Debug.Print oElem("properties")
Next oElem
End Sub
I also get the same error when I try to like the following:
Sub Demo()
Dim Json As Object
JsonString = "[{""Entries"":[{""Name"": ""SMTH"",""Gender"": ""Male""}]}]"
JsonConverter.JsonOptions.AllowUnquotedKeys = True
Set Json = JsonConverter.ParseJson(JsonString)
Debug.Print Json(1)("Entries")
End Sub
I'm on Windows 7 (32 bit) and I'm using this library.
One more thing, they are valid JSON and I didn't encounter any error while parsing the same using Python.
Your code should be failing because oElem("properties") is a dictionary. Furthermore, within that dictionary there are a mixture of datatypes associated with the keys so you will need to test the type and handle appropriately. Or use one of the many readily available programs which will handle that and empty the entire json object for you.
Option Explicit
Public Sub ParseJson()
Dim jsonObject As Object, oElem As Variant
Dim resp$, Url$, R&
Url = "https://torontolife.com/wp-content/themes/sjm-underscores/inc/neighbourhoods/2015/compiled.json"
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", Url, False
.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36"
.send
resp = .responseText
End With
Set jsonObject = JsonConverter.ParseJson(resp)
Dim key As Variant, propertyTypes As Scripting.Dictionary
Set propertyTypes = New Scripting.Dictionary
For Each oElem In jsonObject("features")
For Each key In oElem("properties")
Debug.Print key, vbTab, TypeName(oElem("properties")(key))
propertyTypes(key) = TypeName(oElem("properties")(key))
Next
Next oElem
'Review propertyTypes dict and/or immediate window print out
Stop
End Sub
I'm trying to download files(PDFs) silently from a website with VBA. So far I login without issue entering UserName & Password on the initial screen, navigate to the reports page within the site, get my list of files successfully in a table. I get the URL of the file in question without issue. Here's where I hit a wall. I do download a file but get a security warning when i open it that I must be logged in to view it. I can simulate this warning by pasting a URL into any browser when I'm not logged in & they look the same. So I'm downloading but not authenticating.
The code just on the download issue:
Dim strCookie As String
Dim strResponse As String
Dim xobj As Object
Dim WinHttpReq As Object
Dim WinHttpReq2 As Object
Dim oStream As Object
' Set xobj = New WinHttp.WinHttpRequest
strDocLink = "https://atlasbridge.com" & strDocLink & "&RT=PREVMAIL"
Debug.Print strDocLink
' launch tab & goto url/doc
' try to download the link(this is the url of the file)
' strDocLink
Set WinHttpReq = CreateObject("WINHTTP.WinHTTPRequest.5.1")
strUrl = "https://atlasbridge.com/search/AgencyReports.aspx"
WinHttpReq.Open "GET", strUrl, False
WinHttpReq.Option(WinHttpRequestOption_EnableRedirects) = False
WinHttpReq.setRequestHeader "Referer", "https://atlasbridge.com/search/AgencyReports.aspx"
WinHttpReq.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
WinHttpReq.setRequestHeader "Connection", "keep-alive"
WinHttpReq.setRequestHeader "Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
WinHttpReq.setRequestHeader "Accept-Language", "en-US,en;q=0.5"
WinHttpReq.Send
If WinHttpReq.Status = 200 Then
strResponse = WinHttpReq.responseText
Debug.Print strResponse
strCookie = WinHttpReq.getResponseHeader("Set-Cookie") ' this only gets the cookie; cookie seems include the session id
resp = WinHttpReq.getAllResponseHeaders
' resp = WinHttpReq.responseBody
' strCookie = WinHttpReq.getResponseHeader("Cookie") ' doesnt find the requested header
Debug.Print strCookie
Debug.Print resp
End If
' then open second session & try to get document
Set WinHttpReq2 = CreateObject("WINHTTP.WinHTTPRequest.5.1")
WinHttpReq2.Open "GET", strDocLink, False
WinHttpReq2.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; rv:11.0) like Gecko"
WinHttpReq2.setRequestHeader "Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
WinHttpReq2.setRequestHeader "Accept-Language", "en-US,en;q=0.5"
WinHttpReq2.setRequestHeader "Referer", "https://atlasbridge.com/search/AgencyReports.aspx"
WinHttpReq2.setRequestHeader "Connection", "keep-alive"
WinHttpReq2.setRequestHeader "Host", "atlasbridge.com:443" '
WinHttpReq2.setRequestHeader "Accept-Encoding", "gzip, deflate, br"
' WinHttpReq2.setRequestHeader "Transfer-Encoding", "chunked"
' doesnt like this one causes error on the .send
WinHttpReq2.setRequestHeader "Cache-Control", "private"
WinHttpReq2.setRequestHeader "Upgrade-Insecure-Requests", "1"
WinHttpReq2.setRequestHeader "Content-Type", "application/pdf"
WinHttpReq2.setRequestHeader "Cookie", strCookie
WinHttpReq2.Send
If WinHttpReq2.Status = 200 Then
Set oStream = CreateObject("ADODB.Stream")
oStream.Open
oStream.Type = 1
oStream.Write WinHttpReq2.responseBody
oStream.SaveToFile "C:\Users\MyUserName\Desktop\DownloadedMail\atlasreportdownload.ashx.pdf", 1 ' 1 = no overwrite, 2 = overwrite
oStream.Close
End If
I've tried a few different things, but I don't believe I'm getting the full cookie & session ID.
The cookie I get back in WinHttpReq.getResponseHeader("Set-Cookie") or getAllResponseHeaders looks like:
NSC_bumbtcsjehf.dpn_TTM_443_MCWT=ffffffffc3a00a0a000000000005e445a4a423660;Version=1;Max-Age=2400;path=/;secure;httponly
But when I use LiveHeaders in Firefox I see:
Cookie: ASP.NET_SessionId=z2e4adilfjgiyynx2mntnh1k; NSC_bumbtcsjehf.dpn_TTM_443_MCWT=ffffffffc3a00a0a000000000005e445a4a423660; AuthToken=0be22946-a97a-442e-bd93-c80f0c96a525; AtlasLastMessage=1173; lc_sso7549731=1546651094987; __lc.visitor_id.7549731=S1546651090.26728e19e6
But I can't seem to expose that full cookie with AuthToken & Session ID, etc. when I Debug.Print the response. Can someone point me in the right direction so I can test a variation on what I'm doing? Thank you in advance.
Update: The response headers from the first request:
Cache-Control: private
Date: Wed, 16 Jan 2019 22:04:54 GMT
Content-Length: 164
Content-Type: text/html; charset=utf-8
Location: /default.aspx?err=Expired&dest=%2fhome.aspx
Server: Microsoft-IIS/7.0
Set-Cookie: ASP.NET_SessionId=mo0owzztbul5of0litxox5kx; path=/; secure; HttpOnly
Set-Cookie: NSC_bumbtcsjehf.dpn_TTM_443_MCWT=ffffffffc3a00a1a45525d5f4f58455e445a4a423660;Version=1;Max-Age=2400;path=/;secure;httponly
X-AspNet-Version: 4.0.30319
X-UA-Compatible: IE=edge
X-Powered-By: ASP.NET
I'm working on downloading the response body now.
I am currently attempting to retrieve a json string using XMLHTTP in VBA from the website url detailed below. Loading the first url creates a session, which I retrieve from the HTML body. A call to the second url, using the session ID & other request headers visible from develop tools results in a 403 error. I have tried multiple combinations of headers with no effect. For deployment purposes a VBA solution is required. Any input/ideas would be much appreciated.
Sub test()
Dim wbk_TB As Workbook
Dim var_array As Variant
Dim url As String
Dim data As Variant
Dim XMLHTTP As MSXML2.XMLHTTP
Dim hdoc As MSHTML.HTMLDocument
Set wbk_TB = ThisWorkbook
Set XMLHTTP = New MSXML2.XMLHTTP
url = "http://www.eex-transparency.com/homepage/power/germany/production/availability/non-usability"
XMLHTTP.Open "GET", url, False
XMLHTTP.setRequestHeader "Accept", "application/json, text/plain, */*"
XMLHTTP.send
data = XMLHTTP.responseText
Dim HTMLdoc As MSHTML.HTMLDocument
Set HTMLdoc = New MSHTML.HTMLDocument
HTMLdoc.body.innerHTML = XMLHTTP.responseText
Name = "session=" & HTMLdoc.getElementsByName("session").Item(0).Value
url = "http://www.eex-transparency.com/dsp/tem-12?country=de&limit=50&offset=50"
XMLHTTP.Open "GET", url, True
XMLHTTP.setRequestHeader "Host", "www.eex-transparency.com"
XMLHTTP.setRequestHeader "Proxy-Connection", "keep-alive"
XMLHTTP.setRequestHeader "Accept", "application/json, text/plain, */*"
XMLHTTP.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"
XMLHTTP.setRequestHeader "Referer", "http://www.eex-transparency.com/homepage/power/germany/production/availability/non-usability"
XMLHTTP.setRequestHeader "Accept-Encoding", "gzip,deflate,sdch"
XMLHTTP.setRequestHeader "Cache-Control", "max-age=0"
XMLHTTP.setRequestHeader "Accept-Language", "en-US,en;q=0.8"
XMLHTTP.setRequestHeader "Cookie", Name
XMLHTTP.send
While XMLHTTP.readyState <> 4
DoEvents
Wend
data = XMLHTTP.responseText
End Sub
XMLHttp object does not allow unsafe header settings including spoofed referer header. Details are available in this answer
As the referer header is missing in the request, a status 403 is returned. In case you need to get the JSON from VBA, you would need to use an Internet Explorer object and browse to the first URL and once that is loaded, need to navigate to the Second URL by programatically emulating a click on the correct link and then try to capture the data.
I've tried this so far:
Dim wreq As HttpWebRequest = WebRequest.Create("http://www.nasdaq.com/symbol/goog/financials?query=income-statement&data=quarterly")
wreq.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5"
wreq.Method = "get"
Dim prox As IWebProxy = wreq.Proxy
prox.Credentials = CredentialCache.DefaultCredentials
Dim document As New HtmlAgilityPack.HtmlDocument
Dim web As New HtmlAgilityPack.HtmlWeb
web.UseCookies = True
web.PreRequest = New HtmlAgilityPack.HtmlWeb.PreRequestHandler(AddressOf onPreReq)
wreq.CookieContainer = cookies
Dim res As HttpWebResponse = wreq.GetResponse()
document.Load(res.GetResponseStream, True)
Debug.WriteLine(document.DocumentNode.OuterHtml)
But it returns the HTML for this web address: http://www.nasdaq.com/symbol/goog/financials?query=income-statement
Instead of this one: http://www.nasdaq.com/symbol/goog/financials?query=income-statement&data=quarterly
What am I doing wrong?
Additional Info
Here is onPreReq
Private Function onPreReq(req As HttpWebRequest)
req.CookieContainer = cookies
Return True
End Function
The problem was with my variable Ticker. It contains the ticker symbols that I use to create the web addresses. I have been using all caps (ex. GOOG) up to this point. I've changed all of my tickers to lower case (ex. goog) and that seems to have done the trick.