scrape html without id in vba - html

Question

I'm trying to get month-to-date and year-to-date return values from the website
http://us.spindices.com/indices/equity/sp-oil-gas-exploration-production-select-industry-index
into an Excel spreadsheet using VBA. The problem is that there is no "id= " in the code of the page, which I understand would make this process a lot simpler. There is also the matter of which time period (year-to-date or month-to-date) is visible, but I'd be happy with scraping just the MTD values for now.
Here is my code:
Sub Get_Change()
'attempting to scrape Barclay's website
Dim appIE As Object
Dim MyVar As String
Set appIE = CreateObject("internetexplorer.application")
With appIE
.Navigate "http://us.spindices.com/indices/equity/sp-oil-gas-exploration-production-select-industry-index"
.Visible = True
End With
Do While appIE.Busy
DoEvents
Range("A1").Value = "Working..."
Loop
Set TDelements = appIE.document.getElementsbyClassName("performance-chart-table")
For Each TDelement In TDelements
If TDelement.class = "change" Then
MyVar = TDelement.class.innerText("Value")
End If
Next
Range("A1").Value = MyVar
appIE.Quit
Set appIE = Nothing
End Sub
If I can get a way to set the 'MyVar' variable to the current MTD or YTD value, I'll be done, but I'm having a hard time since there is not a unique identifier for either of these values. Any ideas?

Answer 1

I've recently watched some CSS training videos and I can tell you the CSS selector syntax is powerful and I'd recommend it. This is the same syntax that javascript/web developers use to select elements when using JQuery.
I think you should try using
document.queryselectorall
or in your case because you have drilled in to the document to get the "performance-chart-table" call queryselectorall off of that variable, TDelements.
Documentation at http://www.w3schools.com/jsref/met_document_queryselectorall.asp
and you supply as a parameter a CSS selector string the syntax of which can be found at http://www.w3schools.com/cssref/css_selectors.asp
And I've gone and done it for you....
Sub Get_Change()
'* Tools-References Microsoft HTML Object Library
'attempting to scrape Barclay's website
Dim appIE As Object
Dim MyVar As String
Set appIE = CreateObject("internetexplorer.application")
With appIE
.Navigate "http://us.spindices.com/indices/equity/sp-oil-gas-exploration-production-select-industry-index"
.Visible = True
End With
Do While appIE.Busy
DoEvents
Range("A1").Value = "Working..."
Loop
Dim htmlDoc As MSHTML.HTMLDocument
Set htmlDoc = appIE.document
Dim TDelements2 As MSHTML.IHTMLElementCollection
Set TDelements2 = htmlDoc.getElementsByClassName("performance-chart-table")
While TDelements2.Length < 1
DoEvents
Application.Wait (Now() + TimeSerial(0, 0, 3))
Set TDelements2 = htmlDoc.getElementsByClassName("performance-chart-table")
Wend
Dim oHTMLTablePerformanceChartTable As MSHTML.HTMLTable
Set oHTMLTablePerformanceChartTable = TDelements2.Item(0)
Dim objChangeCollection As MSHTML.IHTMLDOMChildrenCollection
Set objChangeCollection = oHTMLTablePerformanceChartTable.querySelectorAll(".change")
'Debug.Assert objChangeCollection.Length = 2
Dim objChange2 As Object
Set objChange2 = objChangeCollection.Item(1)
MyVar = objChange2.innerText
'Set TDelements = appIE.document.getElementsByClassName("performance-chart-table")
'
'For Each TDelement In TDelements
' TDelements.querySelectorAll (".change")
' If TDelement.class = "change" Then
' MyVar = TDelement.class.innerText("Value")
'
' End If
'Next
Range("A1").Value = MyVar
appIE.Quit
Set appIE = Nothing
End Sub