Render HTML on headless server to produce screenshots - html

I would like to create screenshots of web pages from a given URL. While it's possible to use tools like Selenium RC, that requires a graphical environment. I am running a headless Gentoo server.
This will be part of a tool chain that works like:
Fetch URL
Render HTML
Export render as image file
Store image file

You can run an application with framebuffer X-Server like xvfb - one simple approach is a Qt based app to render the page in a webkit widget and save as an image. Here's a blog post outlining how this can be done with Python.
Here's a quick command line tool I've used with Qt. It's a while since I used it but it should still work!
#include <QtCore/QCoreApplication>
#include <QtGui>
#include <QtWebKit>
#include <QTextStream>
#include <QSize>
QWebView *view;
QString outfile;
void QWebView::loadFinished(bool ok)
{
QTextStream out(stdout);
if (!ok) {
out << "Page loading failed\n";
return;
}
view->page()->setViewportSize(view->page()->currentFrame()->contentsSize());
QImage *img = new QImage(view->page()->viewportSize(), QImage::Format_ARGB32);
QPainter *paint = new QPainter(img);
view->page()->currentFrame()->render(paint);
paint->end();
if(!img->save(outfile, "png"))
out << "Save failure\n";
QApplication::quit();
return;
}
int main(int argc, char *argv[])
{
QTextStream out(stdout);
if(argc < 3) {
out << "USAGE: " << argv[0] << " <url> <outfile>\n";
return -1;
}
outfile = argv[2];
QApplication app(argc, argv);
view = new QWebView();
view->load(QUrl(argv[1]));
return app.exec();
}

Related

Issue with communication between ESP8266 and Arduino UNO

I am having an odd issue. I am providing my codes but I believe the problem is in the wiring which I will explain in a second.
ESP8266 CODE:
#include <SoftwareSerial.h>
SoftwareSerial s(12,14);
#include <ArduinoJson.h>
void setup() {
s.begin(9600);
Serial.begin(9600);
}
void loop() {
StaticJsonBuffer<1000> jsonBuffer;
JsonObject& root = jsonBuffer.createObject();
root["data1"] = 100;
root["data2"] = 200;
Serial.println(s.available());
if(s.available()>0)
{
root.printTo(s);
}
delay(1000);
}
ARDUINO UNO CODE:
#include <SoftwareSerial.h>
#include <ArduinoJson.h>
SoftwareSerial s(5,6);
void setup() {
// Initialize Serial port
Serial.begin(9600);
s.begin(9600);
//while (!Serial) continue;
Serial.println("START");
}
void loop() {
StaticJsonBuffer<1000> jsonBuffer;
JsonObject& root = jsonBuffer.parseObject(s);
if (root == JsonObject::invalid()){
Serial.println("JSON invalid");
Serial.println(s.available());
return;
}
Serial.println("JSON received and parsed");
Serial.print("Data 1: ");
int data1=root["data1"];
Serial.println(data1);
Serial.print("Data 2: ");
int data2=root["data2"];
Serial.println(data2);
delay(1000);
}
So the serial monitor of Arduino Uno (after I plug in both arduino and esp8266) says that s.available() = 0. Therefore, it is not recieving JSON. The weird thing is, when i take out the cables from the pins in Arduino, put them in TX and RX pins and then put them back to pins 5 and 6 everything works. When I unplug the arduino and/or ESP and plug them back in the problem repeats- s.availalbe() = 0 (until I do that weird manouver with taking out the cables and putting them back in). I believe there is something I do not understand wiring-wise. I tried connecting them (esp and uno) to the common GND but it still does not work. Does anyone have any ideas what I might be missing here?
Thanks
PS, thats the setup: wiring pic
Okay, I have just managed to fix the problem. I had to change the pins to 0 and 1 like that:
SoftwareSerial s(0,1);
However, this pins are used to communicate with your computer as well, resulting in the arduino ide not being able to upload the code to our arduino while this pins are occupied. Simple solution was to just upload the code and connect NodeMcu ESP8266 afterwards. Hope this helps somone.

Sphinx Integration in Qt

I would like to integrate Sphinx documentation functionality to help with my Qt project. However, when including the HTML files for Sphinx, the formatting appears differently and no file links work. For example:
QFile file("/home/user1/project/Sphinx/build/html/intro.html");
if (!file.open(QIODevice::Readonly))
qDebug() << "Didn't open file";
QTextStream in(&file);
ui->textBrowser->setText(in.readAll());
Error: QTextBrowser: No document for _sources/intro.txt
This will cause the textBrowser to open the correct file, but will not end up displaying the page with the correct HTML coding, and will not follow the links even though those HTML files are contained in the same path (as I have copied the entire Sphinx project into the Qt project).
Is there some way to package the entire Sphinx project so that inclusion of multiple files is unnecessary or is the multiple file inclusion the way to go and I'm just handling it incorrectly?
Instead of reading all text and setting it with setText() you must use the setSource() method and pass it to the QUrl using the QUr::fromLocalFile() method.
main.cpp
#include <QtWidgets>
class Widget: public QWidget
{
Q_OBJECT
public:
Widget(QWidget *parent=nullptr):
QWidget(parent),
m_text_browser(new QTextBrowser)
{
m_lineedit = new QLineEdit;
auto button = new QPushButton("Load");
auto lay = new QVBoxLayout{this};
auto hlay = new QHBoxLayout;
lay->addLayout(hlay);
hlay->addWidget(m_lineedit);
hlay->addWidget(button);
lay->addWidget(m_text_browser);
connect(button, &QPushButton::clicked, this, &Widget::on_clicked);
}
private slots:
void on_clicked(){
QString fileName = QFileDialog::getOpenFileName(this,
tr("Open Image"),
QDir::homePath(),
tr("HTML Files (*.html)"));
m_lineedit->setText(fileName);
m_text_browser->setSource(QUrl::fromLocalFile(fileName));
}
private:
QTextBrowser *m_text_browser;
QLineEdit *m_lineedit;
};
int main(int argc, char *argv[])
{
QApplication a(argc, argv);
Widget w;
w.showMaximized();
return a.exec();
}
#include "main.moc"

how to parse an html source code

I'm trying to parse a html source code. this is the webpage address I'm trying to parse. I have wrote the code below but it doesn't work at the last step that I wanna pull-out content of meta:
int main(int argc, char *argv[])
{
QApplication a(argc, argv);
QNetworkAccessManager manager;
QNetworkReply *reply = manager.get(QNetworkRequest(QUrl("https://www.instagram.com/p/BTwnRykl6EM/")));
QEventLoop event;
QObject::connect(reply, SIGNAL(finished()), &event, SLOT(quit()));
event.exec();
QString me = reply->readAll();
QString x;
//-------------------------------------------------------------------------------------------------------
//qDebug()<<me;
//-------------------------------------------------------------------------------------------------------
QXmlStreamReader reader(me);
if(reader.readNextStartElement()){
if(reader.name()=="html"){
while (reader.readNextStartElement()) {
if(reader.name()=="head"){
while (reader.readNextStartElement()) {
if(reader.name()=="meta" && reader.attributes().hasAttribute("property") && reader.attributes().value("property").toString()=="og:image")
x = reader.attributes().value("content").toString();
else{
qDebug()<<"why?";
reader.skipCurrentElement();
}
}
}
else
reader.skipCurrentElement();
}
}
else
reader.skipCurrentElement();
}
qDebug()<<x;
return 0;
}
and this part doesn't work:
if(reader.name()=="meta" && reader.attributes().hasAttribute("property") && reader.attributes().value("property").toString()=="og:image")
x = reader.attributes().value("content").toString();
else{
qDebug()<<"why?";
reader.skipCurrentElement();
}
and prints
why?
what is wrong with my code?
HTML is not a valid XML, so you can't use XML parsers. Options for HTML you can find on this wiki page.
Shortly, you can use Qt's Scribe framework or QtWebKit for automatic parsing and rendering HTML, or external libraries for manual parsing:
libxml2 (Win, Mac, Linux)
htmlcxx (Win, Linux)
libhtml (Linux)
libxml2 and libhtml are C libraries, htmlcxx is C++ library, that allows build dom-tree and iterate through it.

C++ program to copy a file into another

I have the following section of code. Am using XCode. Where should i place the source.txt and target.txt so as to be accessible?
std::ifstream fin("source.txt");
std::ofstream fout("target.txt");
fout<<fin.rdbuf(); //sends the text string to the file "target.txt"
system("pause");
return 0;

OpenGL Freeglut 2.8.1 unhandled exception during user callback

I'm learning OpenGl on my own and I'm following: http://www.lighthouse3d.com/tutorials/glut-tutorial/initialization/.
I'm using Visual Studios 2013, and freeglut version 2.8.1, prepackaged: http://www.transmissionzero.co.uk/software/freeglut-devel/
My code is right now simply a copy of the one in the example:
#ifdef __APPLE__
#include <GLUT/glut.h>
#else
#include <GL/glut.h>
#endif
void renderScene(void) {
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
glBegin(GL_TRIANGLES);
glVertex3f(-0.5,-0.5,0.0);
glVertex3f(0.5,0.0,0.0);
glVertex3f(0.0,0.5,0.0);
glEnd();
glutSwapBuffers();
}
int main(int argc, char **argv) {
// init GLUT and create Window
glutInit(&argc, argv);
glutInitDisplayMode(GLUT_DEPTH | GLUT_DOUBLE | GLUT_RGBA);
glutInitWindowPosition(100,100);
glutInitWindowSize(320,320);
glutCreateWindow("Lighthouse3D - GLUT Tutorial");
// register callbacks
glutDisplayFunc(renderScene);
// enter GLUT event processing cycle
glutMainLoop();
return 1;
}
My error:
Unhandled exception at 0x1056131C (ig4icd32.dll) in OpenGL.exe: 0xC000041D: An unhandled exception was encountered during a user callback.
Call Stack:
OpenGL.exe!glutCreateWindow_ATEXIT_HACK(const char * title) Line 616 C++
OpenGL.exe!main(int argc, char * * argv) Line 30 C++
freeglut_std.h(this is the line that my break is happening on):
#define glutInit glutInit_ATEXIT_HACK
static int FGAPIENTRY FGUNUSED glutCreateWindow_ATEXIT_HACK(const char *title) { return __glutCreateWindowWithExit(title, exit); }
I've haven't been able to pinpoint what is causing the error. Some more information, I used Visual Studio's project properties tool to do the linking. The include folder in additional includes, the lib folder in the additional includes. I added the freeglut.lib to additional dependencies. And I dropped the freeglut.dll in the folder with the .exe in the debug folder that Visual Studio creates.
I've already tried #define GLUT_DISABLE_ATEXIT_HACK, which did not work. I'm not really sure where to go from here.
Finally some other info since I have a feeling it might also not be helping.
I'm running windows 8.1, I have Intel integrated chipset family 4, which does not have 8.1 drivers.