I have used link - Parsing JSON to Excel using VBA
to solve my problem, but it is not resolved fully.
Up to JSON Parse it is working as expected then not able to convert it into 2D Array & that's why not able convert JSON data into Excel table.
using code as below,
Option Explicit
Sub GetAPI_Data()
Dim sJSONString As String
Dim sJSONStringTmp1 As String
Dim sJSONStringTmp2 As String
Dim vJSON
Dim sState As String
Dim aData()
Dim aHeader()
' Retrieve JSON content
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://pastebin.com/raw/Zp0mFEqd", True
.send
Do Until .readyState = 4: DoEvents: Loop
sJSONString = "{ ""EmployeeDetails"": " + .responseText + "}"
Debug.Print sJSONString
End With
Debug.Print sJSONString
' Parse JSON sample
JSON.Parse sJSONString, vJSON, sState
If sState = "Error" Then MsgBox "Invalid JSON": End
' Convert JSON to 2D Array
JSON.toArray vJSON("EmployeeDetails"), aData, aHeader
' Output to worksheet #1
Output aHeader, aData, ThisWorkbook.Sheets(1)
MsgBox "Completed"
End Sub
Sub Output(aHeader, aData, oDestWorksheet As Worksheet)
With oDestWorksheet
.Activate
.Cells.Delete
With .Cells(1, 1)
.Resize(1, UBound(aHeader) - LBound(aHeader) + 1).Value = aHeader
.Offset(1, 0).Resize( _
UBound(aData, 1) - LBound(aData, 1) + 1, _
UBound(aData, 2) - LBound(aData, 2) + 1 _
).Value = aData
End With
.Columns.AutoFit
End With
End Sub
My JSON Data as follows,
{
"EmployeeDetails": "[{\"AccountName\":\"CWT COMMODITIES (ANTWERP) N.V.\",\"AccountOwner\":null,\"Age\":\"257\",\"AgreementLevel\":null,\"Amount\":\"1\",\"Amount_converted\":\"1.13\",\"Amount_converted_Currency\":null,\"AmountCurrency\":\"EUR\",\"CloseDate\":\"2022-06-15\",\"CloseMonth\":null,\"CoreTechnology\":null,\"CreatedDate\":\"2021-10-01T07:52:36.000+0000\",\"CustomerIndustry\":\"Infrastructure / Transport\",\"District\":null,\"ePSFBranch_Location\":null,\"ExclusiveHBSTechnology\":null,\"ExpectedProjectDuration\":null,\"FiscalPeriod_Num\":\"6\",\"FiscalYear\":\"2022\",\"ForecastCategory\":\"Pipeline\",\"FPXBranch\":null,\"GrossMargin_Percentage\":null,\"Industry\":\"Education\",\"IndustryCode\":null,\"LeadSource\":null,\"LegacyOpportunityNumber\":null,\"LineofBusiness\":null,\"NextSteps\":null,\"OpportunityName\":\"CWT Onderhoud BRANDDETECTIE\",\"OpportunityOwner\":\"Wim Hespel\",\"OpportunityType\":null,\"OwnerRole\":\"Direct EUR VSK&TTG Sales\",\"PrimarySolutionFamily\":null,\"PrimarySubSolutionFamily\":null,\"Probability_Percentage\":\"5\",\"ProjectEndDate\":\"2022-06-15\",\"ProjectStartDate\":\"2022-06-15\",\"RecordType\":\"Core\",\"Region\":\"Europe\",\"SalesRegion\":\"Belgium & Luxembourg\",\"Stage\":\"1.First Calls\",\"SubRegion\":\"HBS Benelux\",\"OpportunityNumber\":\"0001458471\",\"VerticalMarket\":\"Infrastructure / Transport excluding Airports\",\"Win_LossCategory\":null,\"Win_LossReason\":null,\"Country\":\"Belgium\",\"InitiatedCPQEstimateProcess\":\"False\",\"LastModifiedDate\":\"2022-03-17T15:27:33.000+0000\",\"LocationSS\":null,\"OpportunityCurrency\":null,\"OpportunityID\":\"0065a0000109AMQAA2\",\"OpportunitySubType\":null,\"OwnerID\":\"0051H00000AvuQ2QAJ\",\"RecordTypeId\":\"0121H000001eZ9VQAU\",\"CustomerType\":\"Existing Customer\",\"GBE\":\"HBS\",\"EditedBy\":\"\",\"Field_Or_Event\":\"\",\"OldValue\":\"\",\"NewValue\":\"\",\"EditDate\":\"\",\"LastStageChangeDate\":null,\"StageDuration\":null,\"ExpectedRevenue\":\"0.05\",\"GrossMarginAtSubmission\":null,\"LastActivity\":null,\"OwnerEID\":\"H185118\"}]"
}
Error:
1] on local machine I am getting error in JSON.toArray i.e. not able to create 2D array.
2] while using above code with online JSON Data as per URL then getting only 2 column data which is not proper.
Updated Code
Option Explicit
Sub GetAPI_Data()
Dim sJSONString As String
Dim sJSONStringTmp1 As String
Dim sJSONStringTmp2 As String
Dim vJSON
Dim s
Dim sState As String
Dim aData()
Dim aHeader()
' Retrieve JSON content
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://pastebin.com/raw/Zp0mFEqd", True
.send
Do Until .readyState = 4: DoEvents: Loop
'sJSONString = "{ ""EmployeeDetails"": " + .responseText + "}"
sJSONString = .responseText
Debug.Print sJSONString
End With
' Parse JSON sample
JSON.Parse sJSONString, vJSON, sState
If sState = "Error" Then MsgBox "Invalid JSON": End
Debug.Print vJSON.Item("EmployeeDetails")
'vJSON("EmployeeDetails") = "{ ""EmployeeDetails"": " + vJSON("EmployeeDetails") + "}"
s = vJSON("EmployeeDetails")
s = "{""data"":" & s & "}"
Debug.Print vJSON.Item("EmployeeDetails")
Dim xJSON As Dictionary
'JSON.Parse vJSON("EmployeeDetails"), xJSON, sState
JSON.Parse s, xJSON, sState
If sState = "Error" Then MsgBox "Invalid JSON": End
' Convert JSON to 2D Array
JSON.toArray xJSON, aData, aHeader
' Output to worksheet #1
Output aHeader, aData, ThisWorkbook.Sheets(1)
MsgBox "Completed"
End Sub
Sub Output(aHeader, aData, oDestWorksheet As Worksheet)
With oDestWorksheet
.Activate
.Cells.Delete
With .Cells(1, 1)
.Resize(1, UBound(aHeader) - LBound(aHeader) + 1).Value = aHeader
.Offset(1, 0).Resize( _
UBound(aData, 1) - LBound(aData, 1) + 1, _
UBound(aData, 2) - LBound(aData, 2) + 1 _
).Value = aData
End With
.Columns.AutoFit
End With
End Sub
Note : I have updated API with multiple line of JSON
Error:
1] Now I am getting required data.
2] But the main issue is, it is coming only in 2 rows (1 for column header & other one for Data)
3] Requirement is, it should display 5 different rows with first row of header
Please help me out from this.
This worked for me to give a 2D array which could be placed on a worksheet:
Sub Tester()
Dim json As Object, s As String, recs As Object, arr
Set json = ParseJson(GetContent("C:\Temp\json.txt")) 'reading from a file for testing
s = json("EmployeeDetails") 'get the embedded json
Set json = ParseJson("{""data"":" & s & "}") 'parse the embedded json
Set recs = json("data") 'collection of records 'a Collection of records
arr = RecsToArray(recs) 'convert to a 2D array
With Sheet6.Range("A1")
.Resize(UBound(arr, 1), UBound(arr, 2)).Value = arr 'write array to sheet
End With
End Sub
'Convert an array/collection of json objects (dictionaries)
' to a tabular 2D array, with a header row
Function RecsToArray(recs As Collection)
Dim rec, k, i As Long, r As Long, c As Long, arr()
Dim dictCols As Object
Set dictCols = CreateObject("scripting.dictionary")
i = 0
'Collect all field names (checking every record in case some may be either incomplete or contain "extra" fields)
' Assumes all field names are unique per record, and no nested objects/arrays within a record
For Each rec In recs
For Each k In rec
If Not dictCols.Exists(k) Then
i = i + 1
dictCols.Add k, i
End If
Next k
Next rec
'size the output array
ReDim arr(1 To recs.Count + 1, 1 To i)
'Populate the header row
For Each k In dictCols
arr(1, dictCols(k)) = k
Next k
r = 1
'collect the data rows
For Each rec In recs
r = r + 1 'next output row
For Each k In rec
arr(r, dictCols(k)) = rec(k)
Next k
Next rec
RecsToArray = arr
End Function
Function GetContent(f As String) As String
GetContent = CreateObject("scripting.filesystemobject"). _
OpenTextFile(f, 1).ReadAll()
End Function
The very first issue you have is that you put an additional { "EmployeeDetails" …json… } around your JSON that allready has this
sJSONString = "{ ""EmployeeDetails"": " + .responseText + "}"
Don't do that!
Second issue you have is that you have a string encoded JSON inside a JSON:
So your original JSON is:
{
"EmployeeDetails": "[{\"AccountName\":\"CWT COMMODITIES (ANTWERP) N.V.\",\"AccountOwner\":null,\"Age\":\"257\",\"AgreementLevel\":null,\"Amount\":\"1\",\"Amount_converted\":\"1.13\",\"Amount_converted_Currency\":null,\"AmountCurrency\":\"EUR\",\"CloseDate\":\"2022-06-15\",\"CloseMonth\":null,\"CoreTechnology\":null,\"CreatedDate\":\"2021-10-01T07:52:36.000+0000\",\"CustomerIndustry\":\"Infrastructure / Transport\",\"District\":null,\"ePSFBranch_Location\":null,\"ExclusiveHBSTechnology\":null,\"ExpectedProjectDuration\":null,\"FiscalPeriod_Num\":\"6\",\"FiscalYear\":\"2022\",\"ForecastCategory\":\"Pipeline\",\"FPXBranch\":null,\"GrossMargin_Percentage\":null,\"Industry\":\"Education\",\"IndustryCode\":null,\"LeadSource\":null,\"LegacyOpportunityNumber\":null,\"LineofBusiness\":null,\"NextSteps\":null,\"OpportunityName\":\"CWT Onderhoud BRANDDETECTIE\",\"OpportunityOwner\":\"Wim Hespel\",\"OpportunityType\":null,\"OwnerRole\":\"Direct EUR VSK&TTG Sales\",\"PrimarySolutionFamily\":null,\"PrimarySubSolutionFamily\":null,\"Probability_Percentage\":\"5\",\"ProjectEndDate\":\"2022-06-15\",\"ProjectStartDate\":\"2022-06-15\",\"RecordType\":\"Core\",\"Region\":\"Europe\",\"SalesRegion\":\"Belgium & Luxembourg\",\"Stage\":\"1.First Calls\",\"SubRegion\":\"HBS Benelux\",\"OpportunityNumber\":\"0001458471\",\"VerticalMarket\":\"Infrastructure / Transport excluding Airports\",\"Win_LossCategory\":null,\"Win_LossReason\":null,\"Country\":\"Belgium\",\"InitiatedCPQEstimateProcess\":\"False\",\"LastModifiedDate\":\"2022-03-17T15:27:33.000+0000\",\"LocationSS\":null,\"OpportunityCurrency\":null,\"OpportunityID\":\"0065a0000109AMQAA2\",\"OpportunitySubType\":null,\"OwnerID\":\"0051H00000AvuQ2QAJ\",\"RecordTypeId\":\"0121H000001eZ9VQAU\",\"CustomerType\":\"Existing Customer\",\"GBE\":\"HBS\",\"EditedBy\":\"\",\"Field_Or_Event\":\"\",\"OldValue\":\"\",\"NewValue\":\"\",\"EditDate\":\"\",\"LastStageChangeDate\":null,\"StageDuration\":null,\"ExpectedRevenue\":\"0.05\",\"GrossMarginAtSubmission\":null,\"LastActivity\":null,\"OwnerEID\":\"H185118\"}]"
}
and what you get out of vJSON.Item("EmployeeDetails") is
[
{
"AccountName": "CWT COMMODITIES (ANTWERP) N.V.",
"AccountOwner": null,
"Age": "257",
"AgreementLevel": null,
"Amount": "1",
"Amount_converted": "1.13",
"Amount_converted_Currency": null,
"AmountCurrency": "EUR",
"CloseDate": "2022-06-15",
"CloseMonth": null,
"CoreTechnology": null,
"CreatedDate": "2021-10-01T07:52:36.000+0000",
"CustomerIndustry": "Infrastructure / Transport",
"District": null,
"ePSFBranch_Location": null,
"ExclusiveHBSTechnology": null,
"ExpectedProjectDuration": null,
"FiscalPeriod_Num": "6",
"FiscalYear": "2022",
"ForecastCategory": "Pipeline",
"FPXBranch": null,
"GrossMargin_Percentage": null,
"Industry": "Education",
"IndustryCode": null,
"LeadSource": null,
"LegacyOpportunityNumber": null,
"LineofBusiness": null,
"NextSteps": null,
"OpportunityName": "CWT Onderhoud BRANDDETECTIE",
"OpportunityOwner": "Wim Hespel",
"OpportunityType": null,
"OwnerRole": "Direct EUR VSK&TTG Sales",
"PrimarySolutionFamily": null,
"PrimarySubSolutionFamily": null,
"Probability_Percentage": "5",
"ProjectEndDate": "2022-06-15",
"ProjectStartDate": "2022-06-15",
"RecordType": "Core",
"Region": "Europe",
"SalesRegion": "Belgium & Luxembourg",
"Stage": "1.First Calls",
"SubRegion": "HBS Benelux",
"OpportunityNumber": "0001458471",
"VerticalMarket": "Infrastructure / Transport excluding Airports",
"Win_LossCategory": null,
"Win_LossReason": null,
"Country": "Belgium",
"InitiatedCPQEstimateProcess": "False",
"LastModifiedDate": "2022-03-17T15:27:33.000+0000",
"LocationSS": null,
"OpportunityCurrency": null,
"OpportunityID": "0065a0000109AMQAA2",
"OpportunitySubType": null,
"OwnerID": "0051H00000AvuQ2QAJ",
"RecordTypeId": "0121H000001eZ9VQAU",
"CustomerType": "Existing Customer",
"GBE": "HBS",
"EditedBy": "",
"Field_Or_Event": "",
"OldValue": "",
"NewValue": "",
"EditDate": "",
"LastStageChangeDate": null,
"StageDuration": null,
"ExpectedRevenue": "0.05",
"GrossMarginAtSubmission": null,
"LastActivity": null,
"OwnerEID": "H185118"
}
]
Which you will need to parse again because this still is JSON!
But the converter you use does not accept the JSON to start with [ and thats another issue here. Because if I strip that brackets off so the [ ] in the beginning and end are gone and parse that again it will work:
Sub GetAPI_Data()
Dim sJSONString As String
Dim sJSONStringTmp1 As String
Dim sJSONStringTmp2 As String
Dim vJSON As Dictionary
Dim sState As String
Dim aData()
Dim aHeader()
' Retrieve JSON content
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://pastebin.com/raw/Zp0mFEqd", True
.send
Do Until .readyState = 4: DoEvents: Loop
'sJSONString = "{ ""EmployeeDetails"": " + .responseText + "}" 'don't do this!
sJSONString = .responseText
End With
Debug.Print sJSONString
' Parse JSON sample
JSON.Parse sJSONString, vJSON, sState
If sState = "Error" Then MsgBox "Invalid JSON": End
Debug.Print vJSON.Item("EmployeeDetails")
Dim StripOffOuterBrackets As String
StripOffOuterBrackets = Mid(vJSON.Item("EmployeeDetails"), 2, Len(vJSON.Item("EmployeeDetails")) - 2)
Debug.Print StripOffOuterBrackets
Dim xJSON As Dictionary
JSON.Parse StripOffOuterBrackets, xJSON, sState
If sState = "Error" Then MsgBox "Invalid JSON": End
' Convert JSON to 2D Array
JSON.ToArray xJSON, aData, aHeader
' Output to worksheet #1
Output aHeader, aData, ThisWorkbook.Sheets(1)
MsgBox "Completed"
End Sub
And it outputs the following (and some more lines)
Im using an API to pull data to Excel and JSON parsing to format the data. I have 2 sites from where to get this data. 1 is smaller database with 3000 lines but the other is bigger with hunderds of thousands. Working with the smaller one I get no error and everything works fine but using the same code on bigger database everything juts kind of crashes/hangs up forever.
Code I'm using is surely very faulty and I was told I'm using too many parsings but I'm new to this so can't really figure it out right now.
Code I'm using:
Option Explicit
Sub Times()
Dim sJSONString As String
Dim vJSON
Dim sState As String
Dim aData()
Dim aHeader()
Dim vResult
Dim sName
Dim authKey As String
authKey = "my_auth_key"
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://my_site_url", True
.SetRequestHeader "Authorization", "Bearer " & authKey
.send
Do Until .readyState = 4: DoEvents: Loop
sJSONString = .responseText
End With
JSON.Parse sJSONString, vJSON, sState
If sState = "Error" Then
MsgBox "Invalid JSON"
End
End If
vResult = vJSON("data")
JSON.ToArray vResult, aData, aHeader
With ThisWorkbook.Sheets(1)
.Cells.Delete
.Cells.WrapText = False
OutputArray .Cells(1, 1), aHeader
Output2DArray .Cells(2, 1), aData
.Columns.AutoFit
End With
vJSON.Remove "data"
Application.DisplayAlerts = False
With ThisWorkbook.Sheets
Do Until .Count = 1
.Item(.Count).Delete
Loop
End With
Application.DisplayAlerts = True
For Each sName In vJSON
If IsArray(vJSON(sName)) Or IsObject(vJSON(sName)) Then
JSON.ToArray vJSON(sName), aData, aHeader
With ThisWorkbook.Sheets.Add
OutputArray .Cells(1, 1), aHeader
Output2DArray .Cells(2, 1), aData
.Columns.AutoFit
End With
vJSON.Remove sName
End If
Next
JSON.ToArray vJSON, aData, aHeader
JSON.Parse sJSONString, vJSON, sState
JSON.Flatten vJSON, vResult
JSON.ToArray vResult, aData, aHeader
MsgBox "Completed"
End Sub
Sub OutputArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize(1, UBound(aCells) - LBound(aCells) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
Sub Output2DArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize( _
UBound(aCells, 1) - LBound(aCells, 1) + 1, _
UBound(aCells, 2) - LBound(aCells, 2) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
Maybe by simplifying and correcting the code my bigger database macro would work as well.
The json I am parsing is at this URL https://reqres.in/api/users?page=2. I am using the following code to parse it.
Option Explicit
Sub Test_LateBinding()
Dim objRequest As Object
Dim strUrl As String
Dim blnAsync As Boolean
Dim strResponse As String
Set objRequest = CreateObject("MSXML2.XMLHTTP")
strUrl = "https://reqres.in/api/users?page=2"
blnAsync = True
With objRequest
.Open "GET", strUrl, blnAsync
.SetRequestHeader "Content-Type", "application/json"
.Send
'spin wheels whilst waiting for response
While objRequest.readyState <> 4
DoEvents
Wend
strResponse = .ResponseText
End With
Debug.Print strResponse
End Sub
I can successfully get the json into the strResponse variable. But lets say I want a variable that is equal to "Eve" which is under first name in the json string. How can I set a variable firstName = "Eve" from that json string.
If you need to work with JSON in VBA then I would recommend using this library:
https://github.com/VBA-tools/VBA-JSON
A simple example using that library:
Public Sub Tester()
Dim http As Object, JSON As Object, d
Set http = CreateObject("MSXML2.XMLHTTP")
http.Open "GET", "https://reqres.in/api/users?page=2", False
http.SetRequestHeader "Content-Type", "application/json"
http.Send
Set JSON = ParseJson(http.responseText)
For Each d In JSON("data")
Debug.Print d("id"), d("first_name")
Next
End Sub
Here is VBA example showing how that values could be retrieved. Import JSON.bas module into the VBA project for JSON processing.
Option Explicit
Sub Test()
Dim sJSONString As String
Dim vJSON
Dim sState As String
Dim aItems
Dim firstName As String
Dim oItem
Dim i As Long
Dim aData()
Dim aHeader()
' Retrieve JSON content
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://reqres.in/api/users?page=2", True
.send
Do Until .readyState = 4: DoEvents: Loop
sJSONString = .responseText
End With
' Parse JSON sample
JSON.Parse sJSONString, vJSON, sState
If sState = "Error" Then
MsgBox "Invalid JSON"
End
End If
' Process objects in array
' Get 'data' array of objects, there is no Set keyword for arrays
aItems = vJSON("data")
' Access specific item 'first_name' property
firstName = aItems(0)("first_name")
Debug.Print firstName
' Access each item 'first_name' property
For Each oItem In aItems
firstName = oItem("first_name")
Debug.Print firstName
Next
' Convert array of objects to 2d array
JSON.ToArray aItems, aData, aHeader
' Access each item element with index 1, which corresponds to 'first_name' property
For i = 0 To UBound(aData, 1)
firstName = aData(i, 1)
Debug.Print firstName
Next
' Output 2d array to first worksheet
With ThisWorkbook.Sheets(1)
.Cells.Delete
.Cells.WrapText = False
OutputArray .Cells(1, 1), aHeader
Output2DArray .Cells(2, 1), aData
.Columns.AutoFit
End With
MsgBox "Completed"
End Sub
Sub OutputArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize(1, UBound(aCells) - LBound(aCells) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
Sub Output2DArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize( _
UBound(aCells, 1) - LBound(aCells, 1) + 1, _
UBound(aCells, 2) - LBound(aCells, 2) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
BTW, the similar approach applied in other answers.
I use FIXER.IO popular API in a VBA to get exchange rates into some cells of my worksheet named USD, CNY, INR etc.
Fixer.io API was returning a text format giving the rates I needed.
As of March 6th 2018, the legacy Fixer API (api.fixer.io) was deprecated and changed into a new version that requires an API Access Key (got it upon registration) but only returns a JSON file. If I call the url:
http://data.fixer.io/api/latest?access_key=XXXXXXXXXXXX&symbols=USD,CNY,INR,THB,SGD,AUD
I get this JSON in return:
{"success":true,"timestamp":1523343843,"base":"EUR","date":"2018-04-10","rates":{"USD":1.231986,"CNY":7.757563,"INR":79.980529,"THB":38.462602,"SGD":1.614924,"AUD":1.592345}}
How can I parse the exchange values in my Excel variables (USD, CNY ...) ?
I tried to look around but my very limited programming skill did not help me to adapt any solution. Please give a "for dummy" reply :)
Thanks for any help provided
Regards
Marco
Take a look at the below example. Import JSON.bas module into the VBA project for JSON processing.
Option Explicit
Sub Test()
Dim sJSONString As String
Dim vJSON
Dim sState As String
Dim aData()
Dim aHeader()
' Retrieve data
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "http://data.fixer.io/api/latest?access_key=209f86f5304e0043a0879d8cb45c9c10&symbols=USD,CNY,INR,THB,SGD,AUD", False
.Send
sJSONString = .ResponseText
End With
' Parse JSON response
JSON.Parse sJSONString, vJSON, sState
' Refer to target dictionary containing rates
Set vJSON = vJSON("rates")
' Access to each item in dictionary
Debug.Print vJSON("USD")
Debug.Print vJSON("CNY")
Debug.Print vJSON("INR")
Debug.Print vJSON("THB")
Debug.Print vJSON("SGD")
Debug.Print vJSON("AUD")
' Convert to array and output to worksheet
JSON.ToArray vJSON, aData, aHeader
With Sheets(1)
.Cells.Delete
.Cells.WrapText = False
OutputArray .Cells(1, 1), aHeader
Output2DArray .Cells(2, 1), aData
.Columns.AutoFit
End With
End Sub
Sub OutputArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize(1, UBound(aCells) - LBound(aCells) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
Sub Output2DArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize( _
UBound(aCells, 1) - LBound(aCells, 1) + 1, _
UBound(aCells, 2) - LBound(aCells, 2) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
The output for me as follows:
BTW, the similar approach applied in other answers.
I am working on a project to scrape information from a number of websites. I have a number of sites working with no issue, largely processing them by amending the URL to pass through the relevant criteria or by posting AJAX requests. I am fairly new to this so I am seeking some assistance.
I have come across a website where I need to interact with objects on a page in order to obtain further information. An example of this is the below site:
Example Site
If you visit the site and go to the bottom there are more brands and clicking "view" will display additional products. The HTML for these is only returned once clicked.
With other sites I have sourced information from I have used the below approach. Is there a way to process the page via the XML HTTP Method after a page object action has been performed?
Any help would be greatly appreciated. At the moment I am assuming I will have to stick to scraping such sites using an Internet Explorer object.
Option Explicit
Public Sub sbKF()
Dim conn As ADODB.Connection
Dim rsIn As ADODB.Recordset
Dim HTMLDoc As HTMLDocument
Dim strUrl As String
Dim strPost As String
Set conn = CurrentProject.Connection
Set rsIn = New ADODB.Recordset
Set HTMLDoc = New MSHTML.HTMLDocument
rsIn.Open pcstrInput, conn, adOpenStatic, adLockReadOnly
rsIn.MoveLast: rsIn.MoveFirst
Do While Not rsIn.EOF
' Create the URL and Post submission for input size.
strUrl = "http://www.[Site].com"
strPost = "Stage=2&sop=TyreSize&ssq=1&vnp=&vmk=&vch=&vmo=&drd="
' Return the Document body results
HTMLDoc.body.innerHTML = fnPostXmlHttp(strUrl, strPost)
rsIn.MoveNext
Loop
End Sub
Public Function fnPostXmlHttp(ByVal strUrl As String, ByVal strScript As String)
Dim XMLHttpRequest As Object
Dim strOut As String
Set XMLHttpRequest = CreateObject("MSXML2.XMLHTTP")
XMLHttpRequest.Open "POST", strUrl, False
XMLHttpRequest.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
XMLHttpRequest.send (strScript)
While XMLHttpRequest.ReadyState <> 4
DoEvents
Wend
fnPostXmlHttp = XMLHttpRequest.responseText
End Function
If you take a look at www.blackcircles.com HTML response, you will see the javascript snippet:
...
var newTyresActionUrl;
var lookupAddress;
$(document).ready(function () {
newTyresActionUrl = new BC.classes.productV6SearchPage('https://www.blackcircles.com/order/tyres',
{"Error":false,"VariantFitments":[{"Name":"All Season","VariantType":11,"SeasonalType":true,"TruckType":false,"FriendlyName":"allseason","Count":17,
...
"TakeoverCss":"\u003clink id=\"brandtakeover-css\" rel=\u0027stylesheet\u0027 type=\u0027text/css\u0027 href=\u0027/templates/bcstyles/css/goodyear-effgrip-perf.css\u0027\u003e"},
"Width",
"Profile",
"Rim",
"Speed",
"Method",
true,
""
);
addToBasket = new BC.classes.addtobasket('https://www.blackcircles.com/order/tyres', "order", '/truck/garages');
...
Actually the portion within curly braces represents a JSON object containing all displayed on the webpage data. So you can extract that JSON string from HTML content by Instr(), parse it, convert to arrays and output to the worksheet, as shown in the example code below. Import JSON.bas module into the VBA project for JSON processing.
Option Explicit
Sub Test_blackcircles()
Dim sResp As String
Dim vJSON As Variant
Dim sState As String
Dim i As Long
Dim vItem
Dim aData()
Dim aHeader()
With CreateObject("MSXML2.XMLHTTP")
.Open "GET", "https://www.blackcircles.com/order/tyres/search?width=205&profile=55&rim=R16&speed=V&vehicle-make=&postcode=&delivery=1&findTyre=", False
.send
sResp = .responseText
End With
sResp = getFragment(sResp, "new BC.classes.productV6SearchPage", "new BC.classes.addtobasket")
sResp = getFragment(sResp, "{", "}")
sResp = "{" & sResp & "}"
JSON.Parse sResp, vJSON, sState
i = 1
With Sheets(1)
.Cells.Delete
.Cells.WrapText = False
For Each vItem In Array( _
"Manufacturers", _
"CarManufacturers", _
"All", _
"Deals", _
"Best", _
"Rest", _
"SearchParams" _
)
.Cells(i, 1).Value = vItem
JSON.ToArray vJSON(vItem), aData, aHeader
OutputArray .Cells(i + 2, 1), aHeader
Output2DArray .Cells(i + 3, 1), aData
.Columns.AutoFit
i = i + UBound(aData, 1) + 5
Next
End With
End Sub
Sub OutputArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize(1, UBound(aCells) - LBound(aCells) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
Sub Output2DArray(oDstRng As Range, aCells As Variant)
With oDstRng
.Parent.Select
With .Resize( _
UBound(aCells, 1) - LBound(aCells, 1) + 1, _
UBound(aCells, 2) - LBound(aCells, 2) + 1)
.NumberFormat = "#"
.Value = aCells
End With
End With
End Sub
Function getFragment( _
sourceText As String, _
startPattern As String, _
endPattern As String _
) As String
Dim startPos
startPos = InStr(sourceText, startPattern)
If startPos = 0 Then Exit Function
Dim partText
partText = Mid(sourceText, startPos + Len(startPattern))
Dim endPos
endPos = InStrRev(partText, endPattern)
If endPos = 0 Then Exit Function
getFragment = Left(partText, endPos - 1)
End Function
BTW, the similar approach applied in other answers.