Downloaded page source is different than the rendered page source - html

Question

I'm planning to get data from this website
http://www.gpw.pl/akcje_i_pda_notowania_ciagle
(it's a site of the main stock market in Poland)
I've got a program written in C++ that downloads source of the site to the file.
But the problem is that it doesn't contain thing I'm interested in
(stocks' value of course).
If you compare this source of the site to the option "View element" ( RMB -> View element)
you can see that "View element" does contain the stocks' values.
<td>75.6</td>
<tr class="even red">
etc etc...
The downloaded source of the site doesn't have this information.
So we've got 2 questions
1) Why does source of the site is different from the "View element" option?
2) How to transfer my program so that it can download the right code?
#include <string>
#include <iostream>
#include "curl/curl.h"
#include <cstdlib>
using namespace std;
// Write any errors in here
static char errorBuffer[CURL_ERROR_SIZE];
// Write all expected data in here
static string buffer;
// This is the writer call back function used by curl
static int writer(char *data, size_t size, size_t nmemb,
string *buffer)
{
// What we will return
int result = 0;
// Is there anything in the buffer?
if (buffer != NULL)
{
// Append the data to the buffer
buffer->append(data, size * nmemb);
// How much did we write?
result = size * nmemb;
}
return result;
}
// You know what this does..
void usage()
{
cout <<"curltest: \n" << endl;
cout << "Usage: curltest url\n" << endl;
}
/*
* The old favorite
*/
int main(int argc, char* argv[])
{
if (argc > 1)
{
string url(argv[1]);
cout<<"Retrieving "<< url << endl;
// Our curl objects
CURL *curl;
CURLcode result;
// Create our curl handle
curl = curl_easy_init();
if (curl)
{
// Now set up all of the curl options
curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errorBuffer);
curl_easy_setopt(curl, CURLOPT_URL, argv[1]);
curl_easy_setopt(curl, CURLOPT_HEADER, 0);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer);
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer);
// Attempt to retrieve the remote page
result = curl_easy_perform(curl);
// Always cleanup
curl_easy_cleanup(curl);
// Did we succeed?
if (result == CURLE_OK)
{
cout << buffer << "\n";
exit(0);
}
else
{
cout << "Error: [" << result << "] - " << errorBuffer;
exit(-1);
}
}
}
return 0;
}

Answer 1

Because the values are filled in using JavaScript.
"View source" shows you the raw source for the page, while "View Element" shows you the state the document tree is in at the moment.
There's no simple way to fix it, because you need to either execute the JavaScript or port it to C++ (and it would probably make you unpopular at the exchange).

Answer 2

When I save the page as an html file (file/save as), I get a file containing all data displayed in browser and which was not found in page source (I use Chrome).
So I suggest that you add one step in your code:
Download page from a javascript enabled browser that support command line or some sort of API (If curl can't do it, maybe wget or lynx/links/links2/elinks on linux can help you?).
Parse data.