VBA wait for everything in a webpage to completely load - html

I'm trying to pull some information from a website after navigating to it but I can't seem to wait until it completely loads. I've been trying to loop until the class at (0) contains text. Anyone know what I'm doing wrong?
Sub test()
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
Dim elements2 As IHTMLElementCollection
IE.Navigate "https://www.facebook.com/marketplace/item/955559644646354/"
Do While IE.Busy Or IE.readyState <> 4
DoEvents
Loop
Dim x
x = 0
Do Until x = 1
Set elements2 = IE.document.getElementsByClassName("_3cgd")
If WorksheetFunction.IsText(elements2(0).innerText) = True Then
MsgBox ((elements2(0).innerText))
x = 1
Else
Application.Wait Now + #12:00:01 AM#
End If
Loop
End Sub

Try something like this (untested)
Dim t, elements2, txt
t = Timer
txt = ""
Do
Set elements2 = IE.document.getElementsByClassName("_3cgd")
If elements2.length > 0 Then
txt = elements2(0).innerText
If Len(txt) > 0 Then Exit Do
End If
If (Timer - t) > 10 Then Exit Do 'exit if too long waiting
Application.Wait Now + TimeSerial(0, 0, 1)
Loop

You can try to add lines below to wait for loading the web page completely.
' Wait while IE loading...
'IE ReadyState = 4 signifies the webpage has loaded (the first loop is set to avoid inadvertently skipping over the second loop)
Do While IE.ReadyState = 4: DoEvents: Loop 'Do While
Do Until IE.ReadyState = 4: DoEvents: Loop 'Do Until
Below is the full working example:
Sub Automate_IE_Load_Page()
'This will load a webpage in IE
Dim i As Long
Dim URL As String
Dim IE As Object
Dim objElement As Object
Dim objCollection As Object
'Create InternetExplorer Object
Set IE = CreateObject("InternetExplorer.Application")
'Set IE.Visible = True to make IE visible, or False for IE to run in the background
IE.Visible = True
'Define URL
URL = "https://www.automateexcel.com/excel/"
'Navigate to URL
IE.Navigate URL
' Statusbar let's user know website is loading
Application.StatusBar = URL & " is loading. Please wait..."
' Wait while IE loading...
'IE ReadyState = 4 signifies the webpage has loaded (the first loop is set to avoid inadvertently skipping over the second loop)
Do While IE.ReadyState = 4: DoEvents: Loop 'Do While
Do Until IE.ReadyState = 4: DoEvents: Loop 'Do Until
'Webpage Loaded
Application.StatusBar = URL & " Loaded"
'Unload IE
Set IE = Nothing
Set objElement = Nothing
Set objCollection = Nothing
End Sub
Further, You can try to modify this code example as per your requirement.
Reference:
(1) Automate Internet Explorer (IE) Using VBA

Related

Need Excel VBA to navigate website and download specific files

Trying to understand how to interact with a website in a specific way. This is part of a larger code I'm working on that will loop through a list of ContractorIDs. What I need to do from here is the following:
Navigate to this website: https://ufr.osd.state.ma.us/WebAccess/SearchDetails.asp?ContractorID=042786217&FilingYear=2018&nOrgPage=7&Year=2018
Find the link that says "UFR Filing with Audited Financials" and click on it. (if it's not there, end the sub)
On the ensuing page, find the link that is identified under "Document Category" as "UFR Excel Template" and click on it. (in this case, the link says "15-UFR18.xls", however since there's no consistent link naming scheme, the correct link will always have to be identified by the label under "Document Category" as mentioned. If the link doesn't exist, exit sub.)
On the ensuing page, click the "Download" link at the top and save the file under the following file path (which would be created at this time): C:\Documents\042786217\2018.
Edit: Code below gets me to the point where the download button is clicked, then I get the Open/Save/Cancel dialog box. Nearly there, just need to figure out how to save the file into a specific path.
Option Explicit
Sub UFRScraper()
If MsgBox("UFR Scraper will run now. Do you wish to continue?", vbYesNo) = vbNo Then Exit Sub
Dim IE As Object
Dim objElement As Object
Dim objCollection As Object
Dim ele As Object
Dim tbl_Providers As ListObject: Set tbl_Providers = ThisWorkbook.Worksheets("tbl_ProviderList").ListObjects("tbl_Providers")
Dim FEIN As String: FEIN = ""
Dim FEINList As Range: Set FEINList = tbl_Providers.ListColumns("FEIN").DataBodyRange
Dim ProviderName As String: ProviderName = ""
Dim ProviderNames As Range: Set ProviderNames = tbl_Providers.ListColumns("Provider Name").DataBodyRange
Dim FiscalYear As String: FiscalYear = ""
Dim urlUFRDetails As String: urlUFRDetails = ""
Dim i As Integer
' Create InternetExplorer Object
Set IE = CreateObject("InternetExplorer.Application")
' Show (True)/Hide (False) IE
IE.Visible = True
i = 1
For i = 1 To 3 'Limited to 3 during testing. Change when ready.
FEIN = FEINList(i, 1)
ProviderName = ProviderNames(i, 1)
urlUFRDetails = "https://ufr.osd.state.ma.us/WebAccess/SearchDetails.asp?ContractorID=" & FEIN & "&FilingYear=2018&nOrgPage=1&Year=2018"
IE.Navigate urlUFRDetails
' Wait while IE loading...
'IE ReadyState = 4 signifies the webpage has loaded (the first loop is set to avoid inadvertently skipping over the second loop)
Do While IE.ReadyState = 4: DoEvents: Loop 'Do While
Do Until IE.ReadyState = 4: DoEvents: Loop 'Do Until
'Step 2 is done here
Dim filingFound As Boolean: filingFound = False
For Each ele In IE.Document.getElementsByTagName("a")
If ele.innerText = "UFR Filing with Audited Financials" Then
filingFound = True
IE.Navigate ele.href
Do While IE.ReadyState = 4: DoEvents: Loop 'Do While
Do Until IE.ReadyState = 4: DoEvents: Loop 'Do Until
Exit For
End If
Next ele
If filingFound = False Then
GoTo Skip
End If
'Step 3
Dim j As Integer: j = 0
Dim UFRFileFound As Boolean: UFRFileFound = False
For Each ele In IE.Document.getElementsByTagName("li")
j = j + 1
If ele.innerText = "UFR Excel Template" Then
UFRFileFound = True
IE.Navigate "https://ufr.osd.state.ma.us/WebAccess/documentviewact.asp?counter=" & j - 4
Do While IE.ReadyState = 4: DoEvents: Loop 'Do While
Do Until IE.ReadyState = 4: DoEvents: Loop 'Do Until
Exit For
End If
Next ele
If UFRFileFound = False Then
GoTo Skip
End If
'Step 4
IE.Document.getElementById("LinkButton2").Click
'**Built in wait time to avoid accidentally overloading server with repeated quick requests during development and testing**
Skip:
Application.Wait (Now + TimeValue("0:00:03"))
MsgBox "Loop " & i & " complete."
Next i
'Unload IE
IE.Quit
Set IE = Nothing
Set objElement = Nothing
Set objCollection = Nothing
MsgBox "Process complete!"
End Sub
I have tried step 3 with some what lengthy way. but could not provide complete download code as (after one successful manual attempt) at present even manual download attempt causing massage "The File Could Not Be Retrieved" (maybe server side constrain)
Code only take you down to the cell containing href of the xlx file
Dim doc As HTMLDocument
Dim Tbl As HTMLTable, Cel As HTMLTableCell, Rw As HTMLTableRow, Col As HTMLTableCol
Set doc = IE.document
For Each ele In IE.document.getElementsByClassName("boxedContent")
For Each Tbl In ele.getElementsByTagName("table")
For Each Rw In Tbl.Rows
For Each Cel In Rw.Cells
'Debug.Print Cel.innerText
If InStr(1, Cel.innerText, "UFR Excel Template") > 0 Then
Debug.Print Rw.Cells(2).innerText & " - " & Rw.Cells(2).innerHTML
End If
Next
Next Rw
Next Tbl
Next
Once the href is available PtrSafe Function or WinHTTPrequest or other methods could be used to download the file. Welcome and eager to learn some more efficient answers in this case from experts like #QHarr and others.

How to copy/paste data from excel to a webpage (VBA)?

So I'm trying to copy/paste data from excel into a webpage text box using VBA. However, my problem is that if I have, for example 3 or 4 rows of data copied, when pasting the values into the webpage using vba, only 1 row will be copied rather than all the rows.
Here is my code:
.Document.getElementsByTagName("textarea")(0).Value = ActiveCell.Value
Any ideas? If I take out the (0) I get an error:
object doesn't support this property or method.
This is some code of mine that works:
Sub FillOutInvoice(Account As Account)
Dim ie As InternetExplorer
Dim elem As HTMLLinkElement
Set ie = New InternetExplorer
ie.Visible = True
ie.navigate Settings.ErpAddress
Do While ie.Busy Or ie.readyState <> 4
DoEvents
Loop
With ie.document
.getElementsByClassName(Settings.InputClass)(0).innerText = Account.InvoiceNumber
.getElementsByClassName(Settings.InputClass)(1).innerText = Day(Account.InvoiceDate)
End With
Do While ie.Busy Or ie.readyState <> 4
DoEvents
Loop
Application.Wait Now + #12:00:02 AM#
ie.Quit
End Sub
As you see, the needed property is .InnerText and not .Value.
Here is an example of sending a range of rows text to a textarea element using the clipboard
Option Explicit
'VBE > Tools > References:
' Microsoft Internet Controls
Public Sub InsertData()
Dim ie As New InternetExplorer
With ie
.Visible = True
.Navigate2 "https://www.google.com/search?q=google+translate&rlz=1C1GCEB_enGB815GB815&oq=google+tran&aqs=chrome.0.0j69i57j0l4.2057j0j7&sourceid=chrome&ie=UTF-8"
While .Busy Or .readyState < 4: DoEvents: Wend
Dim clipboard As Object
Set clipboard = CreateObject("New:{1C3B4210-F441-11CE-B9EA-00AA006B1A69}")
ActiveSheet.Range("A1:A3").Copy
With clipboard
.GetFromClipboard
ie.document.getElementsByTagName("textarea")(1).innerText = .GetText
End With
Application.CutCopyMode = False
Stop
.Quit
End With
End Sub
Contents of Range("A1:A3")

VBA Auto Login Web page and fetch the data

I am trying to login Web page and fetch data however my login details are not getting update, i have tried all possibilities code from your forum, nothing is working for me
Below is my code, am getting attached error
Sub test()
Dim ie As Object
Dim objCollection As Object
Dim i As Integer
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
ie.navigate "https://portal.expeditors.com/expo/login"
Do While ie.Busy
Application.Wait DateAdd("s", 1, Now)
Loop
'Get all the elements with input tag name
Set objCollection = ie.document.getElementsByTagName("input")
i = 0
'Loop through all elements and find login form and fill it
While i < objCollection.Length
'Login name
If objCollection(i).Name = "username" Then
objCollection(i).Value = "bom-sumand"
End If
'Store login button in object
If objCollection(i).Type = "submit" Then
Set objElement = objCollection(i)
End If
i = i + 1
Wend
'Click login
objElement.Click
'Clean up
Set ie = Nothing
End Sub
I would use the available ids rather than looping to find the input boxes and sign in. These are much faster selector methods. You can add a .Focus. Also, swop InternetExplorer for InternetExplorerMeduim in some cases.
If problem continues check your internet settings in case site is blocked.
Open the URL via creating an IE instance direct.
Option Explicit
Public Sub Login()
Dim ie As New InternetExplorer 'InternetExplorerMedium
Const MAX_WAIT_SEC As Long = 5
Dim t As Date, ele As Object
With ie
.Visible = True
.navigate "https://portal.expeditors.com/expo/login"
While .Busy Or .readyState < 4: DoEvents: Wend
With .document
Do
DoEvents
On Error Resume Next
Set ele = .getElementById("j_username")
On Error GoTo 0
If Timer - t > MAX_WAIT_SEC Then Exit Do
Loop While ele Is Nothing
If ele Is Nothing Then Exit Sub
With ele
.Focus
.Value = "bob"
End With
With .getElementById("j_password")
.Focus
.Value = "penny"
End With
.getElementById("signInBtn").Click
End With
While .Busy Or .readyState < 4: DoEvents: Wend
Stop '<== Delete me later
.Quit
End With
End Sub
Macro trying and failing to open a second instance of IE try this.
Sub test()
Dim ie As Object
Dim redURL As String
Dim objCollection As Object
Dim i As Integer
redURL = "https://portal.expeditors.com/expo/login"
On Error Resume Next
Set ie = GetObject(, "InternetExplorer.Application")
If Err Then
Set ie = CreateObject("InternetExplorer.Application")
End If
On Error GoTo 0
ie.Visible = True
ie.Navigate redURL
Do While ie.Busy
Loop
End Sub

Excel VBA code to click web button

Need help how to create excel vba code for this
I'll be needing the codes so I can complete my macro.
Thanks in advance
First, you will need to create a reference to:
Microsoft Internet Controls
Microsoft HTML Object Library
In VBE, click Tools > References
Sub clickLink()
Dim ie As New InternetExplorer, Url$, doc As HTMLDocument
Url = "http://UrlToYourLink.com"
With ie
.navigate Url
Do While .Busy Or .readyState < READYSTATE_COMPLETE
DoEvents
Loop
doc = .document
.Visible = True
End With
Dim myBtn As Object
Set myBtn = doc.getElementsByClassName("button rounded")(0)
myBtn.Click
End Sub
The Internet control is used to browse the webpage and the HTML Objects are used to identify the username and password textboxes and submit the text using the control button.
Dim HTMLDoc As HTMLDocument
Dim oBrowser As InternetExplorer
Sub Login_2_Website()
Dim oHTML_Element As IHTMLElement
Dim sURL As String
On Error GoTo Err_Clear
sURL = "https://www.google.com/accounts/Login"
Set oBrowser = New InternetExplorer
oBrowser.Silent = True
oBrowser.timeout = 60
oBrowser.navigate sURL
oBrowser.Visible = True
Do
' Wait till the Browser is loaded
Loop Until oBrowser.readyState = READYSTATE_COMPLETE
Set HTMLDoc = oBrowser.Document
HTMLDoc.all.Email.Value = "sample#vbadud.com"
HTMLDoc.all.passwd.Value = "*****"
For Each oHTML_Element In HTMLDoc.getElementsByTagName("input")
If oHTML_Element.Type = "submit" Then oHTML_Element.Click: Exit For
Next
' oBrowser.Refresh ' Refresh If Needed
Err_Clear:
If Err <> 0 Then
Debug.Assert Err = 0
Err.Clear
Resume Next
End If
End Sub
The program requires references to the following:
1 Microsoft Internet Controls
2. Microsoft HTML Object Library
Microsoft internet controls are a great way to do this, but if you aren't allowed to add new references, here is another way to go about web scraping.
This methode ain't as 'clean' as Microsoft internet controls and HTML object but it gets the job done.
Sub GoogleSearch()
Dim ie As Object
Dim objSearchBnt As Object
Dim objCollection As Object
Dim i As Integer
'initialize counter
i = 0
'Create InternetExplorer Object
Set ie = CreateObject("InternetExplorer.Application")
ie.Visible = True
'navigate to the url
ie.navigate "Www.google.com"
'Statusbar shows in the buttom corner of excel
Application.StatusBar = "Loading, please wait..."
'Wait until page is ready
Do While ie.busy
Application.Wait DateAdd("s", 1, Now)
Loop
'Store all the elements with input tag
Set objCollection = ie.Document.getElementsByTagName("input")
'Go through all input elements
While i < objCollection.Length
'input search field
If objCollection(i).Name = "q" Then
objCollection(i).Value = "Hello World"
End If
'search button
If objCollection(i).Type = "submit" Then
Set objSearchBnt = objCollection(i)
End If
i = i + 1
Wend
objSearchBnt.Click
'Clean up
Set objSearchBnt = Nothing
Set objCollection = Nothing
Set ie = Nothing
'Give excel control over the status bar agian
Application.StatusBar = ""
End Sub

Run-time error number 5 ("Invalid procedure call or argument") when trying to return LocationURL

I'm trying to store the LocationURLproperty of an Internet explorer object but I'm getting a
Run-time error 5 ("Invalid procedure call or argument")
when doing so and can't figure out what the issue is - Here's my code:
Sub MyMacro()
Dim IE As Object
Dim URL As String
Set IE = CreateObject("InternetExplorer.Application")
With sh.Windows(0)
.Visible = True
.navigate "www.google.com"
End With
Do
DoEvents
Loop While IE Is Nothing
Do
DoEvents
Loop While IE.ReadyState <> 4
Do
DoEvents
Loop While IE.Document.ReadyState <> "complete"
URL = IE.LocationURL 'Here's where it crashes
End Sub
If I enter "?IE.LocationURL" in the immediate window and enter, it'll give me the "https://www.google.ie/" that I would expect so I can't see why this can't be assigned to a sting and can't find enough on the error itself to figure it. There's also nothing online that I've been able to find... Can someone advise what I'm doing wrong?!
this works
Option Explicit
Sub MyMacro()
Dim IE As Object
Dim URL As String
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate "www.google.com"
Do
DoEvents
Loop While IE.ReadyState <> 4 Or IE.busy
URL = IE.LocationURL 'Here's where it crashes
Debug.Print URL
IE.Quit
Set IE = Nothing
End Sub