VBScript to recursively scrape a local intranet page for links - csv

Question

I've been tasked with identifying all of the many links we have on our team's intranet. The goal is to declutter (find duplicate links or dead links).
I wrote this script that will go to our page and scrape every link while identifying the file extension. What I'm not sure how to do is to make this recursive. Once it goes to our site and scrapes those links, if it finds another URL (such as htm or html) I want it to follow THAT link and scrape the same from there and continue on until every link associated with the initial URL is exhausted. I'd like it to create a type of hierarchy in the csv such as (example headers):
lvl0_Link_Title,lvl0_File_Type,lvl0_URL,lvl1_Link_Title,lvl1_File_Type,lvl1_URL,lvl2_Link_Title,lvl2_File_Type,lvl2_URL,lvl3_Link...etc.
Obviously, this would end up with a pretty massive csv. If there is a better/cleaner method to achieve the same, I'm open to it.
Set objWshShell = Wscript.CreateObject("Wscript.Shell")
Set fso = CreateObject("Scripting.FileSystemObject")
Set IE = CreateObject("internetexplorer.application")
on error resume next
filename = fso.GetParentFolderName(WScript.ScriptFullName) & "\URL_Dump_Oldsite.csv"
'==============================================
'Create headers for CSV
set output = fso.opentextfile(filename,2,true)
output.writeline "Link Title,File Type,URL"
output.close
'==============================================
IE.Visible = false
IE.Navigate "URL OF OUR INTRANET"
Do While IE.Busy or IE.ReadyState <> 4: WScript.sleep 100: Loop
Do Until IE.Document.ReadyState = "complete": WScript.sleep 100: Loop
for each url in ie.document.getelementsbytagname("a")
if not url.href is nothing then
ext = mid(url.href,instrrev(url.href,"."))
set output = fso.opentextfile(filename,8,true)
output.writeline replace(url.innertext,","," / ") & "," & ext & ",=HYPERLINK(" & chr(34) & url.href & chr(34) & ")"
output.close
end if
next
'===========================================
'Keyword filter for removal
Dim arrFilter
arrFilter = Array("bakpcweb", _
"aims", _
"element", _
"objid", _
"nodeid", _
"objaction", _
"javascript", _
"itemtype")
'===========================================
'Delete lines from csv file containing keywords
strFile1 = fso.GetParentFolderName(WScript.ScriptFullName) & "\URL_Dump_Oldsite.csv"
Set objFile1 = fso.OpenTextFile(strFile1)
Do Until objFile1.AtEndOfStream
i = 0
strLine1 = trim(lcase(objFile1.Readline))
for a = lbound(arrFilter) to ubound(arrFilter)
if instr(strLine1,arrFilter(a)) <> 0 then
i = i + 1
End If
next
if i = 0 then
strNewContents1 = strNewContents1 & strLine1 & vbCrLf
end if
Loop
objFile1.Close
Set objFile1 = fso.OpenTextFile(strFile1,2,true)
objFile1.Write strNewContents1
objFile1.Close
'===========================================
'Delete blank lines from csv file
strFile = fso.GetParentFolderName(WScript.ScriptFullName) & "\URL_Dump_Oldsite.csv"
Set objFile = fso.OpenTextFile(strFile)
Do Until objFile.AtEndOfStream
strLine = objFile.Readline
strLine = Trim(strLine)
If Len(strLine) > 0 Then
strNewContents = strNewContents & strLine & vbCrLf
End If
Loop
objFile.Close
Set objFile = fso.OpenTextFile(strFile,2,true)
objFile.Write strNewContents
objFile.Close
'===========================================
'Remove duplicate lines from csv file
Set objDictionary = CreateObject("Scripting.Dictionary")
strFile = fso.GetParentFolderName(WScript.ScriptFullName) & "\URL_Dump_Oldsite.csv"
Set objFile = fso.OpenTextFile(strFile)
Do Until objFile.AtEndOfStream
strLine = objFile.Readline
strLine = Trim(strLine)
If Not objDictionary.Exists(strLine) Then
objDictionary.Add strLine, strLine
End If
Loop
objFile.Close
Set objFile = fso.opentextfile(strFile,2,true)
For Each strKey in objDictionary.Keys
objFile.WriteLine strKey
Next
objFile.Close
objDictionary.clearall
'===========================================
wscript.echo "Done!"
ie.quit
wscript.quit
Thank you!

Answer 1

This might not be the answer you were expecting, but it sounds like you're reinventing the wheel here, and with a sub-standard tool. In my experience, I also wouldn't find the lvl0, lvl1 etc. format particularly useful when reporting later.
I would strongly recommend you instead use an existing program to scan your intranet, such as Xenu or for a more in-depth analysis, try Screaming Frog SEO Spider (free version is limited to about 500 pages, as I recall, but you can give it a try). These tools have features to save reports, which should suit your needs.
If that doesn't work for you, please comment or edit your answer to explain why you must do this yourself and report in the specified format.
Edit: Here's an example screenshot from the free Xenu program, which lists every resource it tried, its status, links in/out, and type, which can assist with your requirement to report on filetypes. It will also generate full HTML reports if you want stats.