I'm trying to make an executable in VS2008 that will read a webpage source code using a vb.NET function into a string variable. The problem is that the page is not *.html but rather *.aspx.
I need a way to execute the aspx and get the displayed html into a string.
The page I want to read is any page of this type: http://www.realtor.ca/PropertyDetails.aspx?PropertyID=9620716
I have tried the following code, which works properly for html pages, but generates the wrong source code with "access denied" for the page title when I pass in the above aspx page.
Dim myReq As WebRequest = WebRequest.Create(url)
Dim myWebResponse As WebResponse = myReq.GetResponse()
Dim dataStream As Stream = myWebResponse.GetResponseStream()
Dim reader As New StreamReader(dataStream, System.Text.Encoding.UTF8)
Dim responseFromServer As String = reader.ReadToEnd()
Any suggestions or ideas?
I get the same thing while running wget from the command line:
wget http://www.realtor.ca/PropertyDetails.aspx?PropertyID=9620716
I guess the server is relying on that something is set in the browser before the response is delivered, e.g. a cookie. You might want to try using a WebBrowser control (you don't have to have it visible) in the following way (this works):
Public Class Form1
Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
AddHandler WebBrowser1.DocumentCompleted, New WebBrowserDocumentCompletedEventHandler(AddressOf DocumentCompletedHandler)
WebBrowser1.Navigate("http://www.realtor.ca/PropertyDetails.aspx?PropertyID=9620716")
End Sub
Private Sub DocumentCompletedHandler(ByVal sender As Object, ByVal e As WebBrowserDocumentCompletedEventArgs)
Console.WriteLine(WebBrowser1.DocumentText)
End Sub
End Class
Related
I need to open a web page which takes some time to load and then read data from the HTML. I used this :
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.Navigate("Web page which takes some time to load")
Timer1.Interval = 5000 '5s
Timer1.Start()
End Sub
As I was not able to read easily the HTML, I found some code on the web and I added this to convert to HtmlAgilityPack
Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
Dim Doc As New HtmlAgilityPack.HtmlDocument
Doc.LoadHtml(ie.document)
End Sub
But unfortunately, I get the error : the conversion from HTMLDocumentClass to String is not possible.
I tried to use "ieDoc As HtmlDocument" instead of "ie As Object" but with no succes. I tried a lot of examples that I found on the web but I was not able to fix this.
Can someone help me ?
I'm running into a little problem that I haven't found a way to to solve.
I haven't found a forum where this specific problem is addressed, I really hope to find some help.
Here is my code:
Imports System.IO
Imports System.Net
Imports Newtonsoft.Json.Linq
Public Class Form1
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim request As HttpWebRequest
Dim response As HttpWebResponse = Nothing
Dim reader As StreamReader
request = DirectCast(WebRequest.Create("https://pastebin.com/raw/dWjmfW8N"), HttpWebRequest)
response = DirectCast(request.GetResponse(), HttpWebResponse)
reader = New StreamReader(response.GetResponseStream())
Dim jsontxt As String
jsontxt = reader.ReadToEnd()
Dim myJObject = JObject.Parse(jsontxt)
For Each match In myJObject("matches")
Console.WriteLine(match("http")("host").ToString)
Next
End Sub
End Class
Here is the output:
223.16.205.13
190.74.163.58
71.7.168.29
117.146.53.244
31.170.146.28
118.36.122.169
123.7.117.78
113.61.154.182
36.48.37.191
113.253.179.234
124.13.29.41
180.122.74.183
121.157.114.93
39.78.35.216
176.82.1.100
201.143.142.75
222.117.29.229
89.228.209.185
59.153.89.245
148.170.162.37
112.160.243.23
62.101.254.177
190.141.161.149
121.132.177.79
79.165.124.174
118.39.91.43
220.83.82.58
220.161.101.195
190.218.188.86
123.241.174.77
219.71.218.113
81.198.205.2
1.64.205.1
190.204.66.180
203.163.241.36
36.34.148.33
221.124.127.89
115.29.210.231
39.121.63.13
178.160.38.191
117.146.55.217
149.91.99.49
220.93.231.104
49.245.71.40
211.44.70.107
37.119.247.51
222.101.54.200
178.163.102.223
119.198.145.129
188.26.240.141
115.29.233.160
190.164.29.145
94.133.185.144
181.37.196.134
116.88.213.9
115.2.194.11
1.226.12.161
178.63.73.210
49.149.194.242
14.32.29.251
59.0.191.68
58.122.168.43
142.129.230.137
105.145.89.51
201.243.97.65
175.37.162.102
186.88.141.126
105.148.43.100
60.179.173.21
69.115.51.207
90.171.193.132
14.64.76.165
121.127.95.80
175.211.168.48
99.240.74.72
58.153.174.2
119.77.168.142
121.170.47.232
58.243.20.124
199.247.243.234
47.111.76.211
93.72.213.251
218.32.44.73
220.83.90.204
119.158.102.20
95.109.55.204
106.5.19.223
190.199.215.69
190.218.57.249
36.102.72.163
219.78.162.215
177.199.151.96
196.93.125.34
211.58.150.166
180.131.163.40
93.156.97.81
159.89.22.81
130.0.55.156
186.93.202.111
195.252.44.173
What I want to do is to transfer that console output to my Textbox1.Text. Can anyone please show me a way to solve this?
A somewhat simplified method, using WebClient's DownloadStringTaskAsync to download the JSON.
You don't need special treatment here, strings that represent IpAddresses are just numbers and dots and the source encoding is probably UTF8.
After that, just parse the JSON and Select() the property values you care about, transform the resulting Enumerable(Of JToken) to an array of strings and set the array as the source of a TextBox.Lines property.
You can store the lines collection for any other use, in case it's needed.
Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Using client As New WebClient()
Dim json = Await client.DownloadStringTaskAsync([The URL])
Dim parsed = JObject.Parse(json)
Dim lines = parsed("matches").
Where(Function(jt) jt("http") IsNot Nothing).
Select(Function(jt) jt("http")("host").ToString()).ToArray()
TextBox1.Lines = lines
End Using
End Sub
There's no need to transfer anything. If you want the data in a TextBox then put it in a TextBox. You can then output the same data using Console.WriteLine or Debug.WriteLine. You can use a loop:
Dim hosts As New List(Of String)
For Each match In myJObject("matches")
hosts.Add(match("http")("host").ToString())
Next
Dim text = String.Join(Environment.NewLine, hosts)
myTextBox.Text = text
Console.WriteLine(text)
You could also use LINQ:
Dim text = String.Join(Environment.NewLine, myJObject("matches").Select(Function(match) match("http")("host").ToString()))
myTextBox.Text = text
Console.WriteLine(text)
Alternative approach to display collection of things in Winforms are ListView, DataGridView or other collection controls depends on desired usage.
Add ListView control in designer and next code will fill it with received values.
Shared ReadOnly client As HttpClient = New HttpClient()
Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim response As HttpResponseMessage =
Await client.GetAsync("https://pastebin.com/raw/dWjmfW8N")
response.EnsureSuccessStatusCode()
Dim jsonBody As String = Await response.Content.ReadAsStringAsync()
Dim myJObject = JObject.Parse(jsonBody)
ListView1.Items.Clear()
For Each match In myJObject("matches")
ListView1.Items.Add(match("http")("host").ToString)
Next
End Sub
I want to get website's inner text through code.
I can already get it's inner html with code below, but i can't find any code that's getting URL's inner text without webbrowser.
This code is getting text from website in webbrowser, but i need same thing, just without webbrowser.
Dim sourceString As String = WebBrowser1.Document.Body.InnerText
With HtmlAgilityPack...
Private Sub ToolStripButton1_Click(sender As Object, e As EventArgs) Handles ToolStripButton1.Click
Dim doc As HtmlAgilityPack.HtmlDocument = New HtmlAgilityPack.HtmlDocument
With New Net.WebClient
doc.LoadHtml(.DownloadString("https://example.com"))
.Dispose()
End With
Debug.Print(doc.DocumentNode.Name)
PrintChildNodes(doc.DocumentNode)
Debug.Print(doc.DocumentNode.Element("html").Element("body").InnerText)
End Sub
Sub PrintChildNodes(Node As HtmlAgilityPack.HtmlNode, Optional Indent As Integer = 1)
For Each Child As HtmlAgilityPack.HtmlNode In Node.ChildNodes
Debug.Print("{0}{1}", String.Empty.PadLeft(Indent, vbTab), Child.Name)
PrintChildNodes(Child, Indent + 1)
Next
End Sub
**Taken from **
Wolfwyrd
In this question HTTP GET in VB.NET
Try
Dim fr As System.Net.HttpWebRequest
Dim targetURI As New Uri("http://whatever.you.want.to.get/file.html")
fr = DirectCast(HttpWebRequest.Create(targetURI), System.Net.HttpWebRequest)
If (fr.GetResponse().ContentLength > 0) Then
Dim str As New System.IO.StreamReader(fr.GetResponse().GetResponseStream())
Response.Write(str.ReadToEnd())
str.Close();
End If
Catch ex As System.Net.WebException
'Error in accessing the resource, handle it
End Try
You will get Html as well as http headers. Don't think this will work by itself with https.
I have a default page with a button that prompts user to download a "signature"
Which is basically an .html file with specific format (based on user info)
So currently I have an .aspx page but I'm not sure how to make the user download the "rendered HTML page from that aspx"
On the default page i have the following
Protected Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click
Response.ContentType = "text/html"
Response.AppendHeader("Content-Disposition", "attachment; filename=My_Signature.html")
Response.TransmitFile(Server.MapPath("~/Signature.aspx"))
Response.End()
End Sub
Is it possible to render the aspx page in the background then somehow prompt the user to download it ( the resulted html) ?
You're making it more difficult than it is. Simply download the file content as you would from any other website, store it in a string, write it to the response.
Protected Sub Button1_Click(ByVal sender As Object, ByVal e As System.EventArgs) Handles Button1.Click
Response.ContentType = "text/html"
Response.AppendHeader("Content-Disposition", "attachment; filename=My_Signature.html")
Dim contents As String = New System.Net.WebClient().DownloadString(Request.Url.GetLeftPart(UriPartial.Authority) + ResolveUrl("~/Signature.aspx"))
Response.Write(contents)
Response.End()
End Sub
Of course, a better solution would be to put your code for generating the signature in a Class Library (.dll) and then call that as needed.
You could override the Render() method of the aspx file so it writes an html file:
Protected Overrides Sub Render(writer As HtmlTextWriter)
Dim sb As New StringBuilder()
Dim sw As New StringWriter(sb)
Dim hwriter As New HtmlTextWriter(sw)
MyBase.Render(hwriter)
Using outfile As New StreamWriter(Server.MapPath(".") + "\signature.html")
outfile.Write(sb.ToString())
End Using
Response.ContentType = "text/html"
Response.AppendHeader("Content-Disposition", "attachment; filename=signature.html")
Response.TransmitFile(Server.MapPath("~/signature.html"))
Response.End()
End Sub
All this would be in the aspx file to be converted to html (signature.aspx). I would say have your button click do a redirect to a new window that calls the aspx, and thus this method.
i am trying to fetch some data from Url using Httpwebrequest/response, i am getting response which i am showing in a msgbox. It show whole HTML contents.
Now my i want to fetch a specific tag(TD tag) value and show all its value in a combobox in vb.net desktop application.
my code to get response from webserver is :
enter code here
Imports System.IO
Imports System.Net
Imports System.Xml
Imports System.Text.Encoder
Public Class login
Private Sub login_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
End Sub
Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click
End
End Sub
Private Sub Panel1_Paint(ByVal sender As System.Object, ByVal e As System.Windows.Forms.PaintEventArgs) Handles Panel1.Paint
End Sub
Private Sub Ok_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Ok.Click
Dim strId As String = txt_uid.Text
Dim strPwd As String = txt_pwd.Text
Dim oEncoder As New System.Text.ASCIIEncoding
Dim postData As String = "UM_username=" + strId
postData += ("&UM_password=" + strPwd)
Dim data As Byte() = oEncoder.GetBytes(postData)
MsgBox(postData)
Dim webStream As Stream
Dim webResponse As String = ""
Dim req As HttpWebRequest
Dim res As HttpWebResponse
Dim Output As String
'Dim Posit1 As Int32
'Dim Posit2 As Int32
req = WebRequest.Create("http://localhost/basic_framework/index.php?menu=login&UM_email=" & strId & "&UM_password=" & strPwd)
req.ContentType = "application/x-www-form-urlencoded"
req.KeepAlive = False
req.Method = "POST"
res = req.GetResponse()
webStream = res.GetResponseStream()
Dim webStreamReader As New StreamReader(webStream)
While webStreamReader.Peek >= 0
Output = webStreamReader.ReadToEnd()
RichTextBox1.Text = Output
Msgbox(Output)
End While
End Sub
End Class
this code get the response from url and show it in a richtextbox or msgbox
Now i want to get a specific tag value (say, td, option values) and show it in a combobox in my vb.net application form dynamically.It would be needed to parse html content then get that tag value . Please suggest me a way......
If parsing needed , how to parse html contents to get only specific tag value in a combobox in vb.net form
Well...if you can be sure that your request returns valid XHTML (which is XML indeed), you might be able to use an XPath expression.
For the most complicated cases (e.g., an AJAX web-site, etc.) you could use HTMLUnit library with iKVM.