How to extract the div_class table from betting website - html

Question

I was searching continuously and I ran out of luck. I'm trying to just get the text of the games being played on https://www.sportsinteraction.com/football/nfl-betting-lines/. So at the end I want something to spit out like the below (doesn't have to be clean).
Carolina Panthers
Houston Texans
POINTSPREAD
+4.0
1.90
-4.0
1.92
MONEYLINE
2.69
1.49
OVER/UNDER
O
+47.0
1.91
U
+47.0
1.91
Code below which is obviously wrong:
Sub Pulldata2()
Dim ieObj As InternetExplorer
Dim appIE As Object
Dim htmlEle As IHTMLElement
Dim i As Integer
Dim strSheet As String
strSheet = Sheet2.Range("P2")
i = 1
Set ieObj = New InternetExplorer
ieObj.Visible = False
ieObj.navigate Sheet2.Range("P2").Value
Application.ScreenUpdating = False
Application.DisplayStatusBar = False
Application.Wait Now + TimeValue("00:00:03")
Sheet13.Activate
For Each htmlEle In ieObj.document.getElementsByClassName("game")(0)
With ActiveSheet
.Range("A1").Value
End With
i = i + 1
On Error Resume Next
Next htmlEle
End Sub

Answer 1

Why?
This is an interesting scrape for practice so worth the time to attempt. It is interesting because there isn't an immediately obvious way to 'block' out each event with the associated meta data e.g. date, time, match title; and the betting content switches between horizontal and vertical groupings making your selector strategy for identifying nodes particularly important.
Strategy:
The strategy I decided to employ was the following: Use css class selectors to grab a long list of nodes containing all the info required. date, time and title would be nodes whose info I would need to repeat across the two lines for each team within each gameBettingContent block.
Note: the content is static so XHR can be used to avoid the overhead of opening a browser and rendering unnecessary content. This method is a lot quicker.
Anatomy of a logical block:
Note that each parent node with class gameBettingContent contains 3 child nodes with class betTypeContent. These children correspond with PointSpread, MoneyLine and Over/Under. These will need to occupy their own columns in the output including separating out price from handicap. For each logical block, as shown above, there will be two rows where some info is repeated in the first few columns of output.
The nodeList:
So, initially we have a nodeList of all elements of interest matched on className. Within some of those nodes are children that we then need to access; to avoid complicated accessor syntax, when we get to nodes whose children (or deeper nodes) we need to access, we load the html of that node into a 'surrogate', i.e. new, HTMLDocument variable, so we can leverage querySelectorAll method of HTMLDocument again. This gives us easy to read syntax and the cost is minimal.
We loop the nodeList, shown on left above, and test the className of each current node and use a Select Case statement to then determine what to do. If the current className is date, time or title we store that node's .innerText for later use (remember we need to repeat this value across runner 1 and runner 2). If the className is gameBettingContent we load that node's outerHTML into a surrogate HTMLDocument i.e. html2. We can then collect the runners and start populating our output array resultsTable. We add +2 to the r variable (row counter), initially, so we are ready for the next logical block (i.e. match), and then populate the 2 positions for the current match using r-2 and r-1.
Using surrogate HTMLDocument variables to leverage querySelectorAll on nodes deeper in the DOM:
As we are using MSXML2.XMLHTTP with MSHTML.HTMLDocument we do not have access to nth-of-type pseudo class css selector syntax (which you do with ie.document when using SHDocVw.InternetExplorer) to differentiate between the 3 divs (columns) for bet types within gameBettingContent; whilst we could chain class selectors and adjacent sibling combinators to move from left to right across the divs (columns) e.g.
PointSpread:
.gameBettingContent #runnerNames + .betTypeContent
MoneyLine:
.gameBettingContent #runnerNames + .betTypeContent + .betTypeContent
Over/Under:
.gameBettingContent #runnerNames + .betTypeContent + .betTypeContent + .betTypeContent
I find it cleaner to simply gather those divs into a nodeList
Set contentDivs = html2.querySelectorAll(".betTypeContent")
Then load each div's (column's) outerHTML into a new HTMLDocument surrogate html3 and leverage querySelectorAll again to gather the two rows info by index from each column.
For example, PointsSpread will be the first node in contentDivs which we read across into html3:
html3.body.innerHTML = contentDivs.item(0).outerHTML
We then select handicap and price
Set pointSpreadHandicaps = html3.querySelectorAll(".handicap")
Set pointSpreadPrices = html3.querySelectorAll(".price")
And can use index to get runner 1 versus runner 2 values:
resultsTable(r - 2, 5) = pointSpreadHandicaps.item(0).innerText: resultsTable(r - 1, 5) = pointSpreadHandicaps.item(1).innerText
That's pretty much the bulk of the logic. The output array is dimensioned based on number of rows being equal to:
html.querySelectorAll("#runnerNames li").Length
i.e. how many runners there are. The number of columns is equal to the number of items we specified in the headers array (we add 1 to the Ubound as the array is 0-based). The array is then written out in one go to sheet as are headers.
VBA:
Option Explicit
Public Sub GetNFLMatchInfo()
Dim html As HTMLDocument, html2 As HTMLDocument
Set html = New HTMLDocument: Set html2 = New HTMLDocument
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://www.sportsinteraction.com/football/nfl-betting-lines/", False
.send
html.body.innerHTML = .responseText
End With
Dim allNodes As Object, i As Long, resultsTable(), r As Long, headers()
Dim dateValue As String, timeValue As String, title As String, html3 As HTMLDocument
headers = Array("Date", "Time", "Title", "Team", "Pointspread handicap", "Pointspread price", "Moneyline price", "O/U Name", "O/U Handicap", "O/U Price")
Set allNodes = html.querySelectorAll(".date, .time, .title, .gameBettingContent") 'nodeList of all items of interest. gameBettingContent is a block _
that will be further subdivided by reading its html into a 'surrogate' HTMLDocument
ReDim resultsTable(1 To html.querySelectorAll("#runnerNames li").Length, 1 To UBound(headers) + 1)
r = 1: Set html3 = New HTMLDocument
For i = 0 To allNodes.Length - 1
With allNodes.item(i)
Select Case .className
Case "date"
dateValue = .innerText
Case "time"
timeValue = .innerText
Case "title"
title = Trim$(.innerText)
Case "gameBettingContent"
Dim runners As Object, contentDivs As Object, pointSpreadHandicaps As Object
Dim pointSpreadPrices As Object, moneyLinePrices As Object, runners As Object
Dim OuHandicaps As Object, OuPrices As Object
r = r + 2 'then fill line one at r-2, and line 2 at r-1
html2.body.innerHTML = .outerHTML
Set runners = html2.querySelectorAll("#runnerNames li")
resultsTable(r - 2, 1) = dateValue: resultsTable(r - 1, 1) = dateValue
resultsTable(r - 2, 2) = timeValue: resultsTable(r - 1, 2) = timeValue
resultsTable(r - 2, 3) = title: resultsTable(r - 1, 3) = title
resultsTable(r - 2, 4) = runners.item(0).innerText: resultsTable(r - 1, 4) = runners.item(1).innerText
Set contentDivs = html2.querySelectorAll(".betTypeContent")
html3.body.innerHTML = contentDivs.item(0).outerHTML
'populate resultsTable for two rows relating to current gameBettingContent
Set pointSpreadHandicaps = html3.querySelectorAll(".handicap")
Set pointSpreadPrices = html3.querySelectorAll(".price")
resultsTable(r - 2, 5) = pointSpreadHandicaps.item(0).innerText: resultsTable(r - 1, 5) = pointSpreadHandicaps.item(1).innerText
resultsTable(r - 2, 6) = pointSpreadPrices.item(0).innerText: resultsTable(r - 1, 6) = pointSpreadPrices.item(1).innerText
html3.body.innerHTML = contentDivs.item(1).outerHTML 'Set html3 content to next content div to right
Set moneyLinePrices = html3.querySelectorAll(".price")
resultsTable(r - 2, 7) = moneyLinePrices.item(0).innerText: resultsTable(r - 1, 7) = moneyLinePrices.item(1).innerText
html3.body.innerHTML = contentDivs.item(2).outerHTML
Set runners = html3.querySelectorAll(".name")
Set OuHandicaps = html3.querySelectorAll(".handicap")
Set OuPrices = html3.querySelectorAll(".price")
resultsTable(r - 2, 8) = runners.item(0).innerText: resultsTable(r - 1, 8) = runners.item(1).innerText
resultsTable(r - 2, 9) = OuHandicaps.item(0).innerText: resultsTable(r - 1, 9) = .item(1).innerText
resultsTable(r - 2, 10) = OuPrices.item(0).innerText: resultsTable(r - 1, 10) = OuPrices.item(1).innerText
End Select
End With
Next
With ThisWorkbook.Worksheets("Sheet1")
.Cells(1, 1).Resize(1, UBound(headers) + 1) = headers
.Cells(2, 1).Resize(UBound(resultsTable, 1), UBound(resultsTable, 2)) = resultsTable
End With
End Sub
Example selection from results:
Additional reading:
css selectors
querySelectorAll
HTMLDocument