Using Excel VBA to screen-scrape internet - cannot reference second page - html

Question

my question is relatively simple and maddeningly evasive. It's not unlike many questions I've found on the internet and at StackOverflow yet no suggestions have helped my little conundrum....
Using Excel2010 I wasnt to enter data into a single field, submit it (Part1) and capture a few lines of data (part2) , paste into excel in a list/table format (part30 - and do it 999,999 times.....Part 1 and 3 are working - Part2 refuses to acknowledge the new internet window and all gettagnames and SelectTable workarounds just use the original URL - the attached was a desperate attempt using Sendkeys - which worse perfectly! - for the first loop - then absolutely nothing!
anyway, the code should be fairly simple - appologies for some mess in the coding order - it's down to me starting to cut out bits with a scalple but after hours of messing about resorted to hatchets...
Dim HTMLdoc As HTMLDocument
Dim ie As InternetExplorer
Sub EPF_FSA()
'Application.DisplayAlerts = False
Application.EnableEvents = False
Dim iHTML_Element As IHTMLElement
Dim sURL As String
Dim miss1 As Integer
Dim FrmNo As Long
Dim FrmName As String
Dim Address1 As String
Dim Address2 As String
Dim Address3 As String
Dim Address4 As String
Dim Address5 As String
Dim Address6 As String
Dim Address7 As String
Dim Address8 As String
Dim AnyLuck As String
Dim RowNum As Integer
Dim ColNum As Integer
RowNum = 1
ColNum = 1
FrmNo = 100111
While FrmNo <= 100112
'Do While FrmNo <= 100112
On Error GoTo Err_Clear
sURL = "http://www.fsa.gov.uk/register/epfSearchForm.do"
Set ie = CreateObject("internetexplorer.application")
'Set Ex = CreateObject("MicrosoftExcel.application")
ie.navigate sURL
ie.Visible = True
Do
' Wait till the Browser is loaded
Loop Until ie.readyState = READYSTATE_COMPLETE
Set HTMLdoc = ie.document
HTMLdoc.all.epfref.Value = FrmNo
For Each iHTML_Element In HTMLdoc.getElementsByTagName("input")
If iHTML_Element.Type = "submit" Then miss1 = miss1 + 1
If miss1 = 2 Then iHTML_Element.Click: Exit For
Next
Err_Clear:
If Err <> 0 Then Err.Clear
Resume Next
'PART 2 ********************************************************************
Do
' Wait till the Browser is loaded
Loop Until ie.readyState = READYSTATE_COMPLETE
Call SendKeys("^a")
DoEvents
Call SendKeys("^c")
DoEvents
ActiveSheet.PasteSpecial Format:="Text", link:=False, DisplayAsIcon:=False
range("A2").Select
'Copy and select relevant text to sheet 2
Worksheets("Sheet1").Activate
FrmName = Cells(39, "A").Value
Address1 = Cells(59, "A").Value
Address2 = Cells(60, "A").Value
Address3 = Cells(61, "A").Value
Address4 = Cells(62, "A").Value
Address5 = Cells(63, "A").Value
Address6 = Cells(64, "A").Value
Address7 = Cells(65, "A").Value
Address8 = Cells(66, "A").Value
AnyLuck = Cells(47, "A").Value
Worksheets("Sheet2").Activate
Cells(RowNum, "A").Value = FrmNo
Cells(RowNum, "B").Value = FrmName
Cells(RowNum, "C").Value = Address1
Cells(RowNum, "D").Value = Address2
Cells(RowNum, "E").Value = Address3
Cells(RowNum, "F").Value = Address4
Cells(RowNum, "G").Value = Address5
Cells(RowNum, "H").Value = Address6
Cells(RowNum, "I").Value = Address7
Cells(RowNum, "J").Value = Address8
Cells(RowNum, "K").Value = AnyLuck
RowNum = RowNum + 1
'ActiveCell.Offset(1, 0).Select
Worksheets("Sheet1").Activate
Cells.Select
Selection.Delete Shift:=xlUp
range("A2").Select
'MsgBox (FrmNo & Chr(10) & FrmName)
'Part 3
FrmNo = FrmNo + 1
ie.Quit
ie.Quit
Wend
'Loop
Application.EnableEvents = True
End Sub

Answer 1

Looks like you can go directly to the results page. Try:
sUrl = "http://www.fsa.gov.uk/register/epfRefSearch.do?epfRef="
sUrl = sUrl & frmNo
and then just navigate to that page. The actual details are then in a div with an ID of "box"