how to just get the HTML output structure of a site - html

Question

I guess it shows my kookiness here but how do I just get the HTML presentation of a website? for ex., I am trying to retrieve from a Wix site the HTML structure (what is actually being viewed by a user on the screen) but instead I am getting lots of scripts that exist on the site. I am doing a small code test for scraping. Much appreciated.

Answer 1

Alright,here we go. Sorry for the delay.
I used selenium to load the page, that way I could make sure to capture all the markup even if it's loaded by ajax. Make sure to grab the standalone library, that threw me for a loop.
Once the html is retrieved I pass it to jsoup which I use to iterate through the document and remove all the text.
Here's the example code:
// selenium to grab the html
// i chose to use this to get anything that may be loaded by ajax
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
// jsoup for parsing the html
import org.jsoup.Jsoup;
import org.jsoup.parser.Parser;
import org.jsoup.helper.Validate;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.nodes.TextNode;
import org.jsoup.select.Elements;
import java.io.IOException;
public class Example {
public static void main(String[] args) {
// Create a new instance of the html unit driver
// Notice that the remainder of the code relies on the interface,
// not the implementation.
WebDriver driver = new FirefoxDriver();
// And now use this to visit stackoverflow
driver.get("http://stackoverflow.com/");
// Get the page source
String html = driver.getPageSource();
Document doc = Jsoup.parse(html, "", Parser.xmlParser());
for (Element el : doc.select("*")){
if (!el.ownText().isEmpty()){
for (TextNode node : el.textNodes())
node.remove();
}
}
System.out.println(doc);
driver.quit();
}
}
Not sure if you wanted to get rid of the attribute tags as well, currently they are left. However, it's easy enough to modify the code so that some or all of the attribute tags are removed too.

Answer 2

If you just require the content from the page, you can just use ?_escaped_fragment_ on every url to get the static content.
_escaped_fragment_ is an standard approach used for Ajax crawling for crawling the pages which are dynamic in nature or are generated / rendered at client side.
Wix based websites support _escaped_fragment.