I am trying to request a website's html code and use it in an app in Xcode (Swift 3.0) and the pod Alamofire. In the html code online, the data contents that I want to scrape are in a div class that returns data from an Events calendar, in the form of a javascript web plugin. Since the website is not static, when I request the html and print the resulting response as a string, the data I want is not contained in the string. A message appears that says:
<noscript>Your browser must support JavaScript to view this content.
Please enable JavaScript in your browser settings then try again.
Events calendar powered by Trumba
</noscript>
My code using Alamofire looks like:
func downloadCalendar(){
Alamofire.request(urlString).responseString { (AlamofireResponse) in
print(AlamofireResponse.result.value!)
}
}
The urlString is a variable for the actual webpage's url.
Is there a way to get all of the html that appears in the html online into Xcode using Alamofire? If it's not possible with Alamofire is there another way to do this using Swift?
I've tried to accomplish a similar thing, unfortunately to no avail...
It seams AlamoFire grabs the first response it gets....
There is a workaround - use UIWebView:
static let webView = UIWebView()
self.webView.loadRequest(URLRequest.init(url: URL.init(string:"http://example.com")!)
DispatchQueue.main.asyncAfter(deadline: .now()+10.0) {[unowned self] in
if let html = self.webView.stringByEvaluatingJavaScript(from: "document.documentElement.outerHTML")
{print(html)}
}
Where 10.0 is the approx number of seconds required for javascript to finish loading the webpage data.
However since: it's not thread safe, you must use a singleton webView,
import UIKit and can't do it in the background - it's far from the perfect solution...
It might be easier to setup a proxy webserver in between to do the parsing for you.
Cheers!
Related
I am trying to fetch live gold prices from a website using this code.
import 'package:http/http.dart' as http;
import 'package:html/dom.dart' as dom;
import 'package:html/parser.dart' as parser;
getWebData() async {
final response = await http.get(Uri.parse(
"https://www.mcxindia.com/en/market-data/get-quote/FUTCOM/GOLD/05FEB2021"));
dom.Document document = parser.parse(response.body);
print(document);
var element = document.getElementById("litPrice");
}
I know how to fetch text when it's between <a> tag or <p> tag but in this case, it's just between a <span> tag which looks like
<span id="litPrice">47526</span> I am unable to figure out how to get this number.
html pub dev
http pub dev
The method would be the same as in the case of the <a> and <p> tags. The text between tags is available via element.innerHtml
I think the issue you're running into is that the tag you're selecting isn't present in the document returned from that url. In the browser, that element is inserted into the document via javascript after the document is loaded.
There are a couple of ways you could deal with this:
Use an API - If the service you're trying to access provides a REST API, then you can see if the data you're trying to access is available via that route. This is by far the better way, if it's available.
Use a headless browser - This is a really heavy-weight solution, and likely won't work for your use case, but I'll put it here as one of the possibilities anyway. In dart, you can use the puppeteer library to interact with a headless chrome browser. This way, you can load the page, wait for the scripts to run, then scrape the DOM - you should then find the element you're looking for.
When I try to load url everything works fine. I also tried to load normal HTML and that worked fine too.
But this html I got from API is so weird and without tags. Is it possible to somehow present this in webView?
This is how my HTML looks like:
Code to load it :
var html: String
html = cards[Int(slider.value)-1].content
print(html)
webView.loadHTMLString(html, baseURL: nil)
The API is returning the data URL-encoded. So you have two options:
Preferred: Try fetching the data non URL-encoded. Maybe by passing the accept header "text/html", assuming the API might default to "text/plain". Check the API documentation and your request headers.
As a fallback: Use NSString.removingPercentEncoding.
I am new in Angular
What I am going to try is to get the HTML of a page and reproduce it into an iFrame (it is an exercise).
I am using the following piece of code:
var prova = this._http.get(myUrl, {responseType: "text"}).subscribe((x) =>{
console.log(x);
});
I did it on a website (if is needed I can also insert the name of the pages) and it returns the html only of some pages.
In the other case the string x is empty.
Could it depend on connection?
Or there is some way to wait the end of the get request?
Or simply is wrong my approach and I should make a different type of request?
Your most likely going to need to use a library like puppeteer if you want to render a page properly. Puppeteer is a node library and useless headless chrome so I am not sure how well you could really integrate with Angular.
https://github.com/GoogleChrome/puppeteer
I have a page that has a "prev" and "next" day button, and rather than reload the entire page, I simply do an ajax call to a controller that returns the entire partial view, which I replace the div with.
$("#divId").html(ajaxResponse);
Pretty simple.
However, I'm finding that this partial view is vastly more data than I need (html doesn't change at all, just the data) and it's causing slowness on mobile browsers.
My question is, is there a tool out there that will let me return a JSON representation of the model data and refresh all the values on the page automatically?
For example, say I have:
#Html.InputFor(x => x.FirstName)
and the JSON returns
{ FirstName: 'Henry', LastName: 'McLeery' }
Is there a library available that can automate the process of doing:
$("#FirstName").val(ajaxResponse.FirstName);
$("#LastName").val(ajaxResponse.LastName);
etc...
?
Take a look at Angular.js. Angular is a JavaScript framework which uses the mvc pattern.
After binding UI elements to your model the displayed data changes automatically when updating your model. Angular offers a nice api to consume ajax requests with json data.
Look at this:
Get and update json using angular.js
I have a problem receiving and opening a picture via AJAX.
If I call the following page:
http://127.0.0.1:8889/ex?sql=SELECT+Image+FROM+Persons+WHERE+Number+Like+%27%2501%27
a picture is displayed from a blob field in IE8.
Now I would like to open this into a div after someone pressed a key (using AJAX)?
Trying to use xhr.responseText does not work (I get an error. Using it on a text response works). So it seems that my problem is to grab the result from the ajax request.
How can I do this?
Some code and the error message:
var picReturn = xhr.responseText;
=> Could not continue due to the following error: c00ce514
You have three options:
Place the resultant data in an iframe. Not very practical.
Take the result and place it in am image source as a data:uri. Not supported in older browsers and limited to 32/64Kb depending on the browser.
Skip the AJAX and write a web service and use that as your url. This is the best option.
You don't say what language you're using server-side but you essentially want to open a web response, set the header to "image/jpeg" and return your stream.