Trouble with automated download from secure website - ms-access

I'm looking for some assistance in downloading files from a secure website. I don't have much knowledge in this area so I'm not sure what I can provide that will be helpful. If you can tell me what to grab from the website I will.
Here are the basics. I must use IE. I'm using vba to send the automated code. I can login to the website, navigate to their reports page, run my report. It returns a table with a hyperlink, name etc. I'm only at the stage of trying to download just one example so I know the download works. I think I'll be able to loop the table & feed hyperlinks to the loop without a problem.
When I fed the download code its first hyperlink, I got my pdf but upon opening it it is only a message saying you must be logged in to view. Well I was logged in. That used
Call URLDownloadToFile(0, "https://atlasbridge.com/reports/atlasreportdownload.ashx?N=49837661&RT=PREVMAIL", "C:\Users\JCarney\Desktop\DownloadedMail\atlasreportdownload.ashx.pdf", 0, 0)
then I tried:
Dim theURL As String
theURL = "https://atlasbridge.com/reports/atlasreportdownload.ashx? N=49837661&RT=PREVMAIL"
Debug.Print theURL
Dim WinHttpReq As Object
Set WinHttpReq = CreateObject("Microsoft.XMLHTTP")
WinHttpReq.Open "GET", theURL, False, UName, Pword
WinHttpReq.Send
theURL = WinHttpReq.responseBody
If WinHttpReq.Status = 200 Then
Debug.Print theURL
Set oStream = CreateObject("ADODB.Stream")
oStream.Open
oStream.Type = 1
oStream.Write WinHttpReq.responseBody oStream.SaveToFile "C:\Users\JCarney\Desktop\DownloadedMail\atlasreportdownload.ashx.pdf", 2 ' 1 = no overwrite, 2 = overwrite
oStream.Close
End If
The result is the same, a pdf downloads but says I'm not logged in. UName, Pword are string variables that look up the user name & password earlier in the sub. They are looking up the correct info. I'm at a loss as to where to go from here. Can I make vba "click" the hyperlink & download & name the file? or do I continue trying to download from WinHttpReq.
here is what the html of the table looks like:
'<table align="center" id="ctl00_cphMain_dgMailings" style="border-collapse:collapse;" border="1" rules="all" cellspacing="0">
'<td>Policy</td><td>Insured</td><td>Type</td><td>Mailed</td>
' <tbody><tr class="blacktextbold">
' <td> </td><td>Policy</td><td>Insured</td><td>Type</td><td>Mailed</td>
' </tr><tr class="blacktext">
' </tr><tr class="blacktext" style="background-color:#D9D9D9;">
' <td>View</td><td>1501-1603-1276</td><td>SUSAN</td><td>Amended Dec Page</td><td>12/08/2018</td>
' </tr><tr class="blacktext">href="javascript:__doPostBack('ctl00$cphMain$dgMailings$ctl19$ctl01','')">></a></td>
' </tr>
'</tbody></table>
What the properly formatted hyperlink will be, as if you did this manually is:
"https://atlasbridge.com/reports/atlasreportdownload.ashx?N=49837661&RT=PREVMAIL"
The table is showing us the truncated path.
Thank you in advance.
quick edit. When I inspect the element, the "view" hyperlink in the table. I get:
View
If that's helpful. Doesn't shed any light for me but maybe it's helpful.
This is the response text if i use SetCredentials "username","password",, HTTPREQUEST_SETCREDENTIALS_FOR_SERVER for the login screen of the website
login screen webpage
here's the response text when i hit the pdf hyperlink directly:
pdf hyperlink

From your last post, I can see that your manual steps are working.
But, In your code you are using WinHttpReq.Open and other code. Which will not work exactly as you did manually.
I suggest you to create an object of Internet Explorer and than try to use it to open your web page. than you can try to find the PDF file links using its id and try to click the buttons or links to download it.
Example code:
Sub demo()
Dim IEapp As Object
Dim WebUrl As String
Set IEapp = CreateObject("InternetExplorer.Application")
WebUrl = "http://www.microsoft.com"
With IEapp
.Silent = True
.Visible = True
.Navigate WebUrl
Do While .Busy
DoEvents
Loop
Do While .readyState <> 4
DoEvents
Loop
End With
End Sub
Add reference to Microsoft Internet Controls and Microsoft HTML Object Library to run this code.
This is just an example code, you need to modify it and add more code to fulfill your requirement.

Related

How to access the Web using VBA? Please check my code

In order to improve the repeatitive work, I tried to access the Web site which is using in company using VBA.
So, I made code using VBA. And I checked it could be access the normal site such as google, youtube...
But, I don't know why it could not be access the company site.
VBA stopped this line
Set HTMLDoc = IE_ctrl.document
Thank you in advanced.
And I checked one different things(VBA Local values, type) between Normal and company site.
please check below 2 pictures.
Sub a()
Dim IE_ctrl As InternetExplorer
Dim HTMLDoc As HTMLDocument
Dim input_Data As IHTMLElement
Dim URL As String
URL = "https://www.google.com"
Set IE_ctrl = New InternetExplorer
IE_ctrl.Silent = True
IE_ctrl.Visible = True
IE_ctrl.navigate URL
Wait_Browser IE_ctrl
Set HTMLDoc = IE_ctrl.document
Wait_Browser IE_ctrl
Set input_Data = HTMLDoc.getElementsByClassName("text").Item
input_Data.Click
End Sub
Sub Wait_Browser(Browser As InternetExplorer, Optional t As Integer = 1)
While Browser.Busy
DoEvents
Wend
Application.Wait DateAdd("s", t, Now)
End Sub
Normal site(operating well.)
enter image description here
Company site(operating error.)
enter image description here
You can try the following code. Please read the comments. I can't say anymore because I don't know the page or the html of the page.
Sub a()
'Use late binding for what you need
Dim ie As Object
Dim nodeInputData As Object
Dim url As String
url = "https://www.google.com"
'Use the windows GUID to initialize the Internet Explorer, if you
'want to get access to a company page. This helps if there are
'security rules you can't access over other ways of initializing IE
'This don't work in most cases for pages in the "real" web
'Read here for more infos:
'https://blogs.msdn.microsoft.com/ieinternals/2011/08/03/default-integrity-level-and-automation/
Set ie = GetObject("new:{D5E8041D-920F-45e9-B8FB-B1DEB82C6E5E}")
ie.Visible = True
ie.navigate url
'Waiting for the document to load
Do Until ie.readyState = 4: DoEvents: Loop
'If necessary, if there is dynamic content that must be loaded,
'after the ie reports, loading was ready
'(The last three values are: hours, minutes, seconds)
Application.Wait (Now + TimeSerial(0, 0, 1))
'I don't know your html. If you only want to click a button,
'you don't need a varable
'ie.document.getElementsByClassName("text")(0).Click
'will do the same like
Set nodeInputData = ie.document.getElementsByClassName("text")(0)
nodeInputData.Click
'A short explanation of getElementsByClassName() and getElementsByTagName():
'Both methods create a node collection of all html elements that was found
'by the creteria in the brackets. This is because there can be any number of
'html elements with specified class names or tag names. If, for example,
'3 html elements with the class name "Text" were found, a node collection
'with three elements is created by getElementsByClassName("Text").
'These have the indices 0 to 2, as in an array. The individual elements are
'also addressed via these indices. They are indicated in round brackets.
End Sub

How to fill in html autocomplete fields using vba?

I am writing a macro that enter data into a online calculator and get the computed results
Link: https://www.icao.int/environmental-protection/CarbonOffset/Pages/default.aspx
I am trying to fill the values of "from city/airport" in the website.
Here is my Code:
Sub get_CO2()
Dim ie As Object
Set ie = New InternetExplorerMedium
ie.Navigate "https://applications.icao.int/icec"
Do While ie.ReadyState <> 4: DoEvents: Loop
ie.Visible = True
With ie.Document
.getElementById("passengerNo").Value = 7
.getElementById("select1").selectedIndex = 1
.getElementById("select2").selectedIndex = 1
' .getElementByName("frm1").Value = "XXX"
' .getElementByName("to1").Value = "XXX"
.getElementById("computeByInput").Click
End With
End Sub
It is doesnt work. It shows "object do not support this property or method"
How can i fill in this kind of types? What function can i use? If the airport code is not known, is there anyway to enter the name of the country and just pick to first option showing up?
I have figured out how to do this with vba.
EDIT: complete code requested
iedoc.getElementsByName("frm1")(0).innerText = "somecountry" ' enter string of country
Set li_arr = iedoc.getElementById("ui-id-1").getElementsByTagName("li")
Do While li_arr.Length = 0: DoEvents: Loop ' wait for the drop down menu to come up
li_arr(0).Click ' now click the first option
If using browser I would go with Selenium basic automation as it is easy to trigger the events associated with the input fields. Note I am using the iframe src url direct to avoid having to navigate the iframe containing the form. After installing selenium you need to ensure latest Chrome and Chromedriver with chromedriver.exe in the same folder as the selenium executables. Also, go vbe > tools > references and add a reference to selenium type library.
It is also possible, I think, to do a series POST xmlhttp request provided you can grab and pass on the right cookies.
The lines in code below
.FindElementByCss("#ui-id-1 li.ui-menu-item")
and
.FindElementByCss("#ui-id-2 li.ui-menu-item")
select the first list element in the dropdowns.
The drop down values are generated continuously from JSON repsonse to POST requests as you type. I have sent the entire required airport string in one go to ensure the first item is the desired.
Option Explicit
Public Sub GetInfo()
Dim d As WebDriver
Set d = New ChromeDriver
Const Url = "https://applications.icao.int/icec"
With d
.Start "Chrome"
.get Url
.FindElementByCss("[name=frm1]").SendKeys "AAD"
.FindElementByCss("#ui-id-1 li.ui-menu-item").Click
.FindElementByCss("[name=to1]").SendKeys "MGQ"
.FindElementByCss("#ui-id-2 li.ui-menu-item").Click
.FindElementById("computeByInput").Click
Stop 'delete me later
.Quit
End With
End Sub

vba: How to click on element within iframe

My goal is to click an element within a html Iframe, but nothing worked for me so far. Hope someone can advise how to approach this task correctly as I am running in circles for weeks now.
I have tried to click on a div Id, span title but nothing worked so far. I believe it is because a wrong syntex
Option Explicit
Sub it_will_work()
'make the app work faster?
Application.ScreenUpdating = False
Application.DisplayAlerts = False
'--------------------------------
Dim sht As Worksheet
Set sht = ThisWorkbook.Sheets("Fields") 'my data will be stored here
Dim LastRow As Long
LastRow = sht.Cells(sht.Rows.Count, "A").End(xlUp).Row 'range definition
Dim i As Long 'Will be used for a loop that navigate to different url
For i = 2 To LastRow 'First url starts at row 2 untill the last row
Dim IE As Object 'Internet Explorer declaration
Set IE = CreateObject("InternetExplorer.Application")
IE.Visible = True
IE.navigate sht.Range("A" & i).Value 'My url that I want to navigate to
While IE.readyState <> 4 Or IE.Busy: DoEvents: Wend
Dim Doc As New HTMLDocument 'Will be used for the main html page
Set Doc = IE.document
Doc.getElementById("tab7").Click 'data taht need to be updated is here
'Global workgroup data that will effect the workgroup data(dependency)
Doc.getElementById("mcdResourceGlobalWorkgroup_ddltxt").Value = sht.Range("W" & i).Value
Doc.getElementById("mcdResourceGlobalWorkgroup_ddltxt").Focus
Doc.getElementById("mcdResourceGlobalWorkgroup_ddlimg").Click
'Workgroup dropdown, that need to be choosen within the Iframe:
Doc.getElementById("ResourceWorkgroup").Value = sht.Range("X" & i).Value '1) worgroup that I want to insert
Doc.getElementById("ResourceWorkgroup").Focus
Doc.getElementById("_IB_imgResourceWorkgroup").Click '2) Cliking here will generate dropdown values according the value inserted above
Application.Wait Now + TimeValue("00:00:5") 'before refering to Iframe I let the values to be loaded
'***from this point I have the issue where I try to access Iframe and click on the desired element:***
'Here I declare Iframe
Dim objIFRAME As Object
Set objIFRAME = IE.document.getElementsByTagName("iframe")
Debug.Print TypeName(objIFRAME)
'Here I ask to click on a title within the Iframe where value = X
objIFRAME.getElementsByName("title").Value = sht.Range("X" & i).Value.Click
Next i
Application.DisplayAlerts = True
Application.ScreenUpdating = True
End Sub
After the url loads the following steps should happen:
Click on tab 7 -> this will open the correct tab to work on
inseart value from clumn "W" to "Global workgroup" field
focus on "Global workgroup" field
Click on an image that validate the "Global workgroup" field
(validates the instered value)
inseart value from clumn "X" to "Workgroup" field
focus on "Workgroup" field
Click on image that opens the drop down options, which is generated
according the inserted value to "Workgroup" field
Within the Iframe, Click on the title that is equal to value
which was inserted to "Workgroup" field
I have also tried to use Selenium IDE so I can see how the recorded macro access the Iframe and click the desired elemnt:
Command: Select frame | Target: Index=2
Click | Target: css=span[title="APAC"]
I have tried to mimic the stpes above in VBE, but couldn't find a way to write it properly. I event tried to download & apply selenium driver and run the code using the selenium library but got stuck as well.
Below image is the html code of the Iframe and the desired element I want to click on:
You should be able to use the following syntax
ie.document.querySelector("[id='_CPDDWRCC_ifr']").contentDocument.querySelector("span[title=APAC]").click
With selenium you can use
driver.SwitchToFrame driver.FindElementByCss("[id='_CPDDWRCC_ifr']")
driver.FindElementByCss("span[title=APAC]").click
With your existing tag solution you need to use an index. For example,
objIFRAME(0)
Then querySelector on the contentDocument of that.

Incorrect data returned when web-scraping from Internet Explorer using VBA

I am using VBA code (in MS Access, but this problem should occur regardless of the VBA platform) to scrape specific web pages for particular data:
Option Compare Database
Option Explicit
' Requires references to "Microsoft Internet Controls"
' Requires references to "Microsoft HTML Object Library"
Private mFound As Boolean
Private Sub cmdGetFromIE(BaseURL as string)
Const SND_ALIAS_SYSTEMEXCLAMATION = 8531
Dim SW As SHDocVw.ShellWindows
Dim IE As SHDocVw.InternetExplorer
Dim CtrA As Long
Dim TStart As Single
Dim Doc As MSHTML.HTMLDocument
Dim IncElement As MSHTML.IHTMLElement, TitleElement As MSHTML.IHTMLElement, UserElement As MSHTML.IHTMLElement
' ...
Set SW = New SHDocVw.ShellWindows
If SW.Count > 0 Then
For CtrA = 0 To SW.Count - 1
Set IE = SW.Item(CtrA)
If Left(IE.LocationURL, Len(BaseURL)) = BaseURL Then
TStart = Timer
IE.Refresh
Do Until (IE.ReadyState = READYSTATE_COMPLETE And Not IE.Busy) Or Timer > TStart + 30
DoEvents ' Sleep 1 ' Sleep Windows API call procedure to sleep 1s
Debug.Print IE.LocationName, IE.LocationURL, IE.ReadyState, IE.Busy
Loop
If IE.ReadyState = READYSTATE_COMPLETE And Not IE.Busy Then
Set Doc = IE.Document
Set IncElement = Doc.getElementsByClassName("history-item__title ng-binding").Item
Set TitleElement = Doc.getElementsByClassName("history-item__details ng-binding").Item
Set UserElement = Doc.getElementsByClassName("person-summary__full-name_link font-size-xxl ng-binding").Item
Debug.Print CtrA & ";" & Val(Right(IncElement.innerText, 12)) & ";" & TitleElement.innerText & ";" & UserElement.innerText
' Do stuff with the data...
End If
End If
Next
' Do more stuff with the data...
Else
' ... Do 'No IE open' stuff...
End If
End Sub
My problem is that if I open Navigation Page A, and from there navigate to Data Page B 1, the correct data is returned from that page, but if I then navigate back to Navigation Page A and then navigate to Data Page B 2, which is the same form, but contains different data, this code returns some or all of the same data for Data Page B 2 as was returned from Data Page B 1, despite the data pages being refreshed.
I can even navigate to Navigation Page A and then open Data Page B 1 from it in a new tab, and then go back to the Navigation Page A tab and then open Data Page B 2 in yet another new tab, and I still get the problem that I get some or all of Data Page B 1's data from Data Page B 2.
Data Page B appears to be an Angular-JS - populated stock form, with different data depending upon the URL's data section, however that shouldn't matter, I want the page's data as it stands at the instant I run the procedure, but whether I have my code do an IE.Refresh or not, I still have this problem.
If I try Set Doc = New MSHTML.HTMLDocument:Set Doc = Doc.createDocumentFromUrl(IE.LocationURL, ""), I get a "Permission Denied" error at Doc.getElementsByClassName.
How can I scrape Data Page B for the correct current data for each IE tab?
EDIT:
I don't necessarily even have to use IE to load the pages, I just need to get the URLs from IE, and if there is another way accessible via VBA to load and parse the resultant HTML DOM that will work, I'm open to it.

unable to EXPORT data from current open web page using VBA

I want to automate my delivery status for my regular courier from various service provider like Blue Dart.
I have Docket Numbers; I tried the same using VBA but it is unable to fetch data from webpage.
My code enter the Docket number from cell in home page, then it redirects to other page where delivery status is mentioned in table.
Sub GetCourseList()
Dim IE As Object
Set IE = CreateObject("InternetExplorer.Application")
Dim IEWindows As SHDocVw.ShellWindows
Dim IEwindow As SHDocVw.InternetExplorer
Dim IEDocument As MSHTML.HTMLDocument
Dim BreadcrumbDiv As MSHTML.HTMLElementCollection
Set IEWindows = New SHDocVw.ShellWindows
'create new instance of IE. use reference to return current open IE if
'you want to use open IE window. Easiest way I know of is via title bar.
IE.Navigate "http://www.bluedart.com/maintracking.html"
'go to web page listed inside quotes
IE.Visible = True
While IE.busy
DoEvents 'wait until IE is done loading page.
Wend
IE.Document.All("numbers").Value = ThisWorkbook.Sheets("sheet1").Range("A1")
Application.SendKeys "~"
Dim URL As String
Dim qt As QueryTable
Dim ws As Worksheet
Set ws = Worksheets.Add
For Each IEwindow In IEWindows
If InStr(IEwindow.LocationURL, "your URL or some unique string") <> 0 Then ' Found it
Set IEDocument = IEwindow.Document
URL = IEwindow.LocationURL
Set qt = ws.QueryTables.Add( _
Connection:="URL;" & URL, _
Destination:=Range("F1"))
With qt
.RefreshOnFileOpen = True
.Name = "bluedart"
.FieldNames = True
.WebSelectionType = xlAllTables
.Refresh BackgroundQuery:=False
End With
End If
Next
End Sub
Your code does not attempt to interact with a page generated after entering Docket Number and confirming in any way. It could be done by:
Emulating browser interaction, can be Internet Explorer: click "Go" element on the page after Docket Number has been entered and use:
While IE.Busy Or IE.Readystate <> 4
DoEvents
Wend
It can also be achieved by creating POST request with proper parameters, including Docket Number.
Even after this is achieved, it still won't be possible to get data by query from this page, as its URL is this:
http://www.bluedart.com/servlet/RoutingServlet
Try to open this link. Nothing will display, because content of this URL is generated via POST method and parameters needed to generate content properly are not included in URL.
Instead of query, data can be accessed via finding HTML elements, such as tables, in HTML document for both methods I've mentioned.