I'm writing an app in Swift 3.0 and I'm trying to scrape data from the search results of a webpage. I perform the search by including the search query as a parameter in the url, but the html that's getting returned to me has no results. I believe this is because the ajax on the webpage has not finished and populated the html with the search results by the time the html is returned to my app.
Question
How do I wait for the search results to load before getting the html?
EDIT:
URL: https://uscdirectory.usc.edu/web/directory/faculty-staff/#basic=a
This url performs a search on the USC directory for the character 'a'. The html in my browser on my MacBook includes these tags:
<tbody>(Search results are here)</tbody>
but in my app the html that is returned to me has nothing:
<tbody></tbody>
This is because the webpage initially has no search results, and then some time later the ajax finishes and the table body is populated. How do I use a URLSessionDataTask object in Swift to wait until the ajax finishes and ONLY THEN give me the html, such that I actually get the search results?
The workaround solution that I devised is unpleasant but functional. I created a WKWebView off screen and loaded the webpage I was interested in. I used a Timer to wait some arbitrary number of seconds for the javascript on the page to finish and then I retrieved the html from the webview
You can use property ajaxComplete() to fix this.
Ex:
$(document).ajaxComplete(function(){
//your code to render HTML
});
It will render your HTML only when ajax request is complete.
Related
I'm new to ASP.NET MVC and not that good with coding. I'm trying to download my returned list from view as a PDF document
The exact error I get is:
System.IO.IOException
HResult=0x80131620
The document has no pages.
Source=itextsharp
StackTrace:
at iTextSharp.text.pdf.PdfPages.WritePageTree()
at iTextSharp.text.pdf.PdfWriter.Close()
at iTextSharp.text.pdf.PdfDocument.Close()
at iTextSharp.text.Document.Close()
My controller with function
My view html
Your problem is in your jquery and your html, you are not passing any values into your div, your table is also not in your div. Thus jquery is pulling nothing, that means that your stream is getting no bytes.
I have been trying to extract the html values from a page e.g. https://www.qwant.com/?q=cat&t=web but when I use jSoup or HtmlUnit I always get a basic page that doesn't compare to what is generated when I search via my normal browser.
My codes in general work on other websites but could someone explain to me why when I visit the above with code that I don't get the same results? I am trying to fetch all the url values on the page. Is it to do with javascript?
WebClient wb = new WebClient(BrowserVersion.FIREFOX_52);
wb.getPage(url);
wb.waitForBackgroundJavaScript(25000);
System.out.println(wb.getCurrentWindow().getEnclosedPage().getWebResponse().getContentAsString());
Some website just won't allow you to parse them headlessly (for obvious reasons). As I tried to curl the Qwant cat result page, the result was a blank page.
But you want to give a try at switching from Firefox to Chrome as your browser : It is not possible to detect and block Chrome headless
My page request content data after html loaded, it take 2 second for completed show page.
When i set all data in one html(server side rendering) and size 485KB,
it take 4 second to show the page.
Why a 485KB size html load time slow than a 1.6MB json?
My server:
a self-host console server.
Owin & RazorEngine for html pages,
WebAPI for json data.
My 2 web page:
1.
html, just static file,
use ajax(WebAPI) request json data.
2.
Set all data in html, server side rendering.
--07/27 18:21 edit
*old title: Why small index html slow than a big json
I do some more test:
*All contents finally are all same.
1. html + webapi request content.
2. html + render content at server.
3. html with content,render nothing.
4. use webapi get html with content no gzip.
5. just for compare, stackoverflow has more waiting time, less download time.This is what i want.
Looks like the problem is RazorEngine.
Rendered html download speed slower than static file.
How do i fix this?
--07/28 10:38 edit
Just find out what problem is.
It's not RazorEngine's problem,the problem is gzip.
Some big html(977KB) use gzip(69KB) download speed faster(1.95s) than raw html(7.80s),
but sometimes gziped html download speed slower than raw html (third img).
I set file size over 100kb to use gzip, but download speed still not stable.
Is there other problem i not found?
Because when HTML is loaded, it also needs time for rendering on the browser depending upon the content. Thus, the time taken is more.
I have a page that has a "prev" and "next" day button, and rather than reload the entire page, I simply do an ajax call to a controller that returns the entire partial view, which I replace the div with.
$("#divId").html(ajaxResponse);
Pretty simple.
However, I'm finding that this partial view is vastly more data than I need (html doesn't change at all, just the data) and it's causing slowness on mobile browsers.
My question is, is there a tool out there that will let me return a JSON representation of the model data and refresh all the values on the page automatically?
For example, say I have:
#Html.InputFor(x => x.FirstName)
and the JSON returns
{ FirstName: 'Henry', LastName: 'McLeery' }
Is there a library available that can automate the process of doing:
$("#FirstName").val(ajaxResponse.FirstName);
$("#LastName").val(ajaxResponse.LastName);
etc...
?
Take a look at Angular.js. Angular is a JavaScript framework which uses the mvc pattern.
After binding UI elements to your model the displayed data changes automatically when updating your model. Angular offers a nice api to consume ajax requests with json data.
Look at this:
Get and update json using angular.js
I have a problem receiving and opening a picture via AJAX.
If I call the following page:
http://127.0.0.1:8889/ex?sql=SELECT+Image+FROM+Persons+WHERE+Number+Like+%27%2501%27
a picture is displayed from a blob field in IE8.
Now I would like to open this into a div after someone pressed a key (using AJAX)?
Trying to use xhr.responseText does not work (I get an error. Using it on a text response works). So it seems that my problem is to grab the result from the ajax request.
How can I do this?
Some code and the error message:
var picReturn = xhr.responseText;
=> Could not continue due to the following error: c00ce514
You have three options:
Place the resultant data in an iframe. Not very practical.
Take the result and place it in am image source as a data:uri. Not supported in older browsers and limited to 32/64Kb depending on the browser.
Skip the AJAX and write a web service and use that as your url. This is the best option.
You don't say what language you're using server-side but you essentially want to open a web response, set the header to "image/jpeg" and return your stream.