Using MSXML2.XMLHTTP in access VBA not extracting all page data - html

Question

Currently, we are using below mentioned code for data extraction but code is not extracting complete data from webpage, code is ignoring data which is visible when i enable java script and DOM storage on Internet explorer.
till now I use below mentioned code, trailing code is extracting every thing accept images from web page.
My code is given blow.
Set http = CreateObject("MSXML2.XMLHTTP")
http.Send
html.body.innerHTML = http.ResponseText
On Error GoTo 0
html1 = html.body.innerHTML
brand5 = html.documentElement.innerHTML
If html1 Like "*media__thumb*" Then
other_img = html.getElementsByClassName("media__thumb")(0).innerText
'other_img = other_img.innerHTML
End If
On webpage multiple image html code is given below(please note that my above code is not extracting data from below mentioned html code.
<a class="media__thumbnail" data-media_type="IMAGE" data-media_id="orbit-bagged-53017-64" data-target="IMAGE" data-has-index="true">
<img src="https://images.yourweb/_145.jpg">
</a>
<a class="media__thumbnail media__thumbnail--selected" data-media_type="IMAGE" data-media_id="orbit-bagged-53017-e1" data-target="IMAGE" data-has-index="true">
<img src="https://images.yourweb1_145.jpg">
</a>
</span></a>
http.response is given below
<div id="thumbnails" class="media__thumbnails" data-component="thumbnails"></div>
<script type="text/template" id="media__thumbnails">
{{#thumbnails}}
<a class="media__thumbnail" data-media_type="{{type}}" data-media_id="{{id}}" data-target="{{type}}" data-has-index="true">
<img src="{{{thumb}}}"/>
{{# hasIcon}}
{{# threeSixtyIcon}} <div class="whitespace"><span class="threesixtyIcon"></span></div>{{/ threeSixtyIcon}}
{{^ threeSixtyIcon}} <span class="videoIcon"></span>{{/ threeSixtyIcon}}
{{/ hasIcon}}
</a>
{{/thumbnails}}
{{#additionalThumbnailsThumbnail}}
<a class="media__thumbnail media__thumbnail-additional-count" data-media_type="{{type}}" data-media_id="{{id}}" data-target="{{type}}" data-has-index="true">
<img src="{{{thumb}}}"/>
{{# hasIcon}}
{{# threeSixtyIcon}} <div class="whitespace"><span class="threesixtyIcon"></span></div>{{/ threeSixtyIcon}}
{{^ threeSixtyIcon}} <span class="videoIcon"></span>{{/ threeSixtyIcon}}
{{/ hasIcon}}
{{#additionalImagesCount}}
<div class="media__thumbnail-overlay"></div>
<span class="media__thumbnail-count">+{{additionalImagesCount}}</span>
{{/additionalImagesCount}}
</a>

Answer 1

That content required javascript to run on the page so you will need to either:
Search for the information for those part of those urls in the web traffic to see if you obtain from elsewhere (you can - shown below); or,
Automate a browser e.g. with microsoft internet controls
You can see from the following that the content is loaded dynamically:
<script type="text/template" id="media__thumbnails">
{{#thumbnails}}
<a class="media__thumbnail" data-media_type="{{type}}" data-media_id="{{id}}" data-target="{{type}}" data-has-index="true">
<img src="{{{thumb}}}"/>
{{# hasIcon}}
{{# threeSixtyIcon}} <div class="whitespace"><span class="threesixtyIcon"></span></div>{{/ threeSixtyIcon}}
{{^ threeSixtyIcon}} <span class="videoIcon"></span>{{/ threeSixtyIcon}}
{{/ hasIcon}}
</a>
{{/thumbnails}}
{{#additionalThumbnailsThumbnail}}
......
{{/additionalThumbnails}}
</script>
<script type="text/template"
1) Network tab - different url
Using a different url found in network tab returning json containing links. Response is json so a jsonparser is required.
'VBE>Tools> References> Add reference to Microsoft Scripting Runtime
'Download and add in jsonconverter.bas from https://github.com/VBA-tools/VBA-JSON/blob/master/JsonConverter.bas
Place the converter.bas code in module2 and comment out the line: Attribute VB_Name = "JsonConverter"
In module1 place the GetInfo sub.
Option Compare Database
Option Explicit
Public Sub GetInfo()
Dim json As Object, url1 As String, url2 As String, url3 As String
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://www.homedepot.com/p/svcs/frontEndModel/100001020?_=1556447908065", False
.send
Set json = Module2.ParseJson(.responseText)
End With
'Parse json object (see paths shown below for example)
url1 = json("primaryItemData")("media")("mediaList")(2)("location")
url2 = json("primaryItemData")("media")("mediaList")(3)("location")
url3 = json("primaryItemData")("media")("mediaList")(4)("location") 'example
Stop '<==delete me later
End Sub
Paths to first 3 thumbnails:
json►primaryItemData►media►mediaList►2►location
json►primaryItemData►media►mediaList►3►location
json►primaryItemData►media►mediaList►4►location
Explore json here.
2) Automated browser (IE version):
'VBE > Tools > References:
' Microsoft Internet Controls
Public Sub GetImageLinks()
Dim ie As New InternetExplorer, images As Object, i As Long
With ie
.Visible = True
.Navigate2 "https://www.homedepot.com/p/Orbit-Sandstone-Rock-Valve-Box-Cover-53017/100001020"
While .Busy Or .readyState < 4: DoEvents: Wend
Set images = .document.querySelectorAll(".media__thumbnail img")
For i = 0 To images.Length - 1
Debug.Print images.item(i).src
Next
Stop
.Quit
End With
End Sub