I have to parse html documents having bold text as section identifiers. But bold text is in different forms , some examples shown below .
Using Beautiful soup I am able to parse them but have to write lot of if else to handle different types of bold. Is there an optimal way to find such bold text and the text in between with using so many if else.
<div style="line-height:120%;padding-bottom:12px;font-size:10pt;">
<font style="font-family:inherit;font-size:10pt;font-weight:bold;">List 1. Work</font>
</div>
<td style="vertical-align:top;padding-left:2px;padding-top:2px;padding-bottom:2px;padding-right:2px;">
<div style="text-align:left;font-size:10pt;">
<font style="font-family:inherit;font-size:10pt;font-weight:bold;">List 1.</font>
</div>
</td>
<td style="vertical-align:top;padding-left:2px;padding-top:2px;padding-bottom:2px;padding-right:2px;">
<div style="text-align:left;font-size:10pt;">
<font style="font-family:inherit;font-size:10pt;font-weight:bold;">Work.</font>
</div>
</td>
<p style="font-family:times;text-align:justify">
<font size="2">
<a name="de42901_List_1._Work"> </a>
<a name="toc_de42901_2"> </a>
</font>
<font size="2"><b> List 1. Work <br> </b></font>
</p>
<p style="font-family:times;text-align:justify">
<font size="2">
<a name="da18101_List_1._Work"> </a>
<a name="toc_da18101_3"> </a>
</font>
<font size="2"><b> List 1. </b></font>
<font size="2"><b><i>Work <br> </i></b></font>
</p>
use the split and join function to remove unwanted /n /b /t and  :
soup = BeautifulSoup(data, 'html.parser')
data = soup.find_all('b')
for i in data :
final = ' '.join([x for x in i.text.split()])
print (final)
it will arrange your data in same format, hope it resolves your query
I've been racking my brain trying to make a listing of members in a website that I used tables in the past to be responsive. I want to design a new site with Bootstrap but just can't seem to find the proper way to get the records to format the same way I had it before. Right now it's locked in with 4 across but if I can get the records to fill the available width, that would be ok. Can anyone help?
Here is the code I used in html to display:
<!-- access ppocc_members -->
<?php require_once('Connections/ppocc_members.php'); ?>
<?php
mysql_select_db($database_ppocc_members, $ppocc_members);
$query_ppoccfrontpage = "SELECT id, uploaded_file_name, StudioName, Phone1, `Website URL` FROM form_results WHERE CurrentMember = 'yes' AND form_results.WebsiteAd = 'yes' ORDER BY RAND() ";
$ppoccfrontpage = mysql_query($query_ppoccfrontpage, $ppocc_members) or die(mysql_error());
$row_ppoccfrontpage = mysql_fetch_assoc($ppoccfrontpage);
$totalRows_ppoccfrontpage = mysql_num_rows($ppoccfrontpage);
?>
<!-- end access ppocc_members -->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Test Member Database</title>
</head>
<body>
<p align="left"><font color="#879ECB" size="6" face="Georgia, Times New Roman, Times, serif">W</font><font color="#666666" size="3" face="Georgia, Times New Roman, Times, serif">elcome
to the Professional Photographers of Cape Cod website, where you can
find the most extensive list on the net of over 20 Cape Cod
professional photographers and videographers
specializing <br>
in all areas
of photography as well as videography. <a href="members/findaphotographer_speciality.php">Find
a Photographer</a>, and other
information for <br>
Brides & Grooms, families, seniors and others interested in the finest
photography
offered <br>
by many of Cape Cod's most experienced photographers.<br>
</font><font size="3" face="Georgia, Times New Roman, Times, serif"><br>
<font color="#879ECB" size="2">Click on any image below to get more information about a member studio</font></font><font color="#666699" size="3" face="Georgia, Times New Roman, Times, serif"> </font></p>
<table bgcolor="#FFFFFF" >
<!--DWLayoutTable-->
<tr>
<?php
$ppoccfrontpage_endRow = 0;
$ppoccfrontpage_columns = 4; // number of columns
$ppoccfrontpage_hloopRow1 = 0; // first row flag
do {
if($ppoccfrontpage_endRow == 0 && $ppoccfrontpage_hloopRow1++ != 0) echo "<tr>";
?>
<td width="184" height="218" align="center" valign="top" bgcolor="#FFFFFF"><p align="center"> <img src="members/files/<?php echo $row_ppoccfrontpage['uploaded_file_name']; ?>" alt="Click For More Details" name="StudioImage" border="0"></p>
<p align="center"><a href="members/details2.php?recordID=<?php echo $row_ppoccfrontpage['id']; ?>"><font color="#879ECB" size="1" face="Verdana, Arial, Helvetica, sans-serif"><em><font color="#999999">Studio
Profile</font></em><strong><br>
<?php echo $row_ppoccfrontpage['StudioName']; ?></strong></font></a><a href="details2.php?recordID=<?php echo $row_ppoccfrontpage['id']; ?>"><font color="#000000" size="1" face="Verdana, Arial, Helvetica, sans-serif"><br>
</font></a><font color="#999999" size="1" face="Verdana, Arial, Helvetica, sans-serif"><?php echo $row_ppoccfrontpage['Phone1']; ?><br>
</font><font color="#999999" size="2" face="Verdana, Arial, Helvetica, sans-serif"><a href="<?php echo $row_ppoccfrontpage['Website URL']; ?>" target="_blank"><font size="1"><em>Visit
Website</em><br>
<font color="#000000"><?php echo $row_ppoccfrontpage['StudioName']; ?></font></font></a></font></p>
</td>
<?php $ppoccfrontpage_endRow++;
if($ppoccfrontpage_endRow >= $ppoccfrontpage_columns) {
?>
</tr>
<tr>
<td height="13"></td>
</tr>
<?php
$ppoccfrontpage_endRow = 0;
}
} while ($row_ppoccfrontpage = mysql_fetch_assoc($ppoccfrontpage));
if($ppoccfrontpage_endRow != 0) {
while ($ppoccfrontpage_endRow < $ppoccfrontpage_columns) {
echo("<td> </td>");
$ppoccfrontpage_endRow++;
}
echo("</tr>");
}?>
</table>
</body>
</html>
This is my current website that uses the mysql.
www.ppocc.org
When using Bootstrap you should use the grid system instead of tables.
Bootstrap v4 example:
<div class="row">
<div class="col"></div>
<div class="col"></div>
<div class="col"></div>
<div class="col"></div>
</div>
Here is Bootstraps documentation.
Please note that the classes have changed dramatically in v4, so if you are using another version you should refer to that versions documentation on the grid system.
Consider the below HTML snippet:
String content = "<div id='id_1' contenteditable='false' data-hb='true' data-hbtype='ifequal' data-inv='' data-body='ifequal' data-code='{{#ifequal eventInfo.eventType "customer"}}'>
<td style="line-height:18px;" align="left">
<font face="Trebuchet MS, arial, helvetica" size="1" color="#58595b" style="font-size:13px;">
This is just to confirm that we've cancelled your <span id='id_2' contenteditable='false' data-hb='true' data-hbtype='variable' data-inv='' data-body='' data-code='{{clientInfo.clientName}}'>clientInfo.clientName</span> parcel as requested.
</font>
</td>
</div>"
Here upon doing Jsoup.parse(content) I am getting below HTML content (with pretty print):
<html>
<head></head>
<body>
<div id="id_1" contenteditable="false" data-hb="true" data-hbtype="ifequal" data-inv="" data-body="ifequal" data-code="{{#ifequal eventInfo.eventType "customer"}}">
<font face="Trebuchet MS, arial, helvetica" size="1" color="#58595b" style="font-size:13px;"> This is just to confirm that we've cancelled your <span id="id_2" contenteditable="false" data-hb="true" data-hbtype="variable" data-inv="" data-body="" data-code="{{clientInfo.clientName}}">clientInfo.clientName</span> parcel as requested.</font>
</div>
</body>
</html>
As can be seen, the <td> element is gone in the parsed content. Has anyone faced the same, or am I missing something here?
I also tried https://try.jsoup.org/ which is some sort of a playground to try JSoup, but this too gives the same output.
Is there any other HTML parser I should be using here?
I'm trying to get the html source of a webpage www.chemguide.co.uk (it has pages that aren't miles long) using winsock implemented in c++. Most of the data that comes through is good, but at certain points in the output there is a certain character (it looks like |¦ on the console and some sort of I on here) being repeated, I think in groups of 8, and there are some other strange characters as well.
Also, some of the document seems to be printed after the end of the page (the tag. Here's the code:
// Portprog.cpp : Defines the entry point for the console application.
//
#include "stdafx.h"
#include <winsock2.h>
#include <sys/types.h>
#include <stdio.h>
#include <iostream>
#include <string>
#include <fstream>
#pragma comment(lib, "ws2_32.lib") //Winsock library
int getHTML(std::string *result)
{
WSADATA wsa;
SOCKET s;
SOCKADDR_IN server;
using std::string;
using std::cout;
using std::endl;
cout << "Initialising Winsock...";
if (WSAStartup(MAKEWORD(2, 2), &wsa) != 0)
{
cout << "Failed. Error Code: " << WSAGetLastError();
return 1;
}
cout << "Winsock initialised." << endl;
if ((s = socket(AF_INET, SOCK_STREAM, 0)) == INVALID_SOCKET)
{
cout << "Could not create socket: " << WSAGetLastError() << endl;
return 1;
}
cout << "Socket created." << endl;
server.sin_addr.s_addr = inet_addr("217.27.240.124");
server.sin_family = AF_INET;
server.sin_port = htons(80); //host to network endian short
//Connect to remote server
if (connect(s, (SOCKADDR *)&server, sizeof(server)) < 0)
{
cout << "Connection failed." << endl;
return 1;
}
cout << "Connected." << endl;
//Send some data
string srequest = "GET / HTTP/1.1\r\n";
srequest += "Host: chemguide.co.uk\r\n";
srequest += "Connection: close\r\n";
srequest += "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5\r\n";
srequest += "\r\n";
char crequest[10000];
int requestSize = srequest.length() + 1;
strncpy_s(crequest, srequest.c_str(), requestSize);
if (send(s, crequest, requestSize, 0) < 0)
{
cout << "Data could not be sent." << endl;
return 1;
}
cout << "Data sent." << endl;
//Receive a reply from the server
std::string server_reply = "";
int recv_length;
char buffer[1000];
int i = 0;
do
{
i = recv_length = recv(s, buffer, sizeof(buffer), 0);
server_reply += buffer;
} while (i != 0);
cout << "Reply received." << endl;
*result = server_reply;
closesocket(s);
WSACleanup();
return 0;
}
int main(int argc, char *argv[])
{
std::string response = "";
getHTML(&response);
std::cout << response << std::endl;
std::ofstream file("output.txt");
file << response;
file.close();
return 0;
}
And here's the output:
HTTP/1.1 200 OK
Date: Mon, 03 Aug 2015 00:22:17 GMT
Server: Apache/2.2.11
Last-Modified: Mon, 13 Apr 2015 11:56:25 GMT
ETag: "99190a-1ec2-51399cdaacc40"
Accept-Ranges: bytes
Content-Length: 7874
Connection: close
Content-Type: text/html
<html>
<head>
<title>chemguide: helping you to understand Chemistry - Main Menu</title>
<meta name="description"
content="Main menu of a site aimed to help advanced level chemistry students to understand chemistry" />
<meta name="keywords"
content="chemistry, A'level, a level, a'level, a-level, advanced level, advanced, help, understand, understanding, guide, guidebook" />
</head>
<body bgcolor="#ffffcc" link="blue" vlink="teal" alink="red">
<a name="top"></a>
<center>
<table align="center" border="0" width="480" cellspacing="10">
<tr>
<td colspan="2" bgcolor="#ccffcc" height="50" align="center" valign="middle">
<font color="#006600" size="7" face="Helvetica, Arial"><b>chemguide</b></font></td>
</tr>
<tr>
<td colspan="2">
<font colorÌÌÌÌÌÌÌÌè="#006600" size="6" face="Helvetica, Arial"><p align="center"><b>Helping you to understand Chemistry</b></p></font>
<font color="#000000" size="5" face="Helvetica, Arial">
<p align="center"><b>MAIN MENU</b></p>
</font>
<pre>
</pre>
<table align="center" cellpadding="10" border="1">
<tr valign="top"><td bgcolor="#cccccc"> <font color="#ff0000" face="Helvetica, Arial" size="2"><b>New! </b></a></font><font color="#000000" face="Helvetica, Arial" size="2">stry" />
<meta name="keywords"
content="chemistry, A'level, a level, a'level, a-level, advanced level, advanced, help, understand, understanding, guide, guidebook" />
</head>
<body bgcolor="#ffffcc" link="blue" vlink="teal" alink="red">
<a name="top"></a>
<center>
<table align="center" border="0" width="480" cellspacing="10">
<tr>
<td colspan="2" bgcolor="#ccffcc" height="50" align="center" valign="middle">
<font color="#006600" size="7" face="Helvetica, Arial"><b>chemguide</b></font></td>
</tr>
<tr>
<td colspan="2">
<font colorÌÌÌÌÌÌÌÌÌI have just come across a really good site of short chemistry revision videos. You can find more about it at the top of the </font>links page.</td></tr>
</table>
<pre>
</pre>
<table align="center" cellpadding="10" border="1">
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2"><b>Keyword searching</b></font></td><td><font color="#000000" face="Helvetica, Arial" size="2">I have removed the Google search box because it was giving problems. Follow this link to find out how you can still search Chemguide using keywords.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2"><b>Edexcel Chemistry book</b></font></td><td><font color="#ff0000" face="Helvetica, Arial" size="2"><b>Support pages for my Edexcel International GCSE Chemistry book. This will soon be retitled as Edexcel International GCSE Chemistry, Edexcel Certificate inÌÌÌÌÌÌÌÌè Chemistry.</b></font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2"><b>CIE syllabus support</b></font></td><td><font color="#ff0000" face="Helvetica, Arial" size="2"><b>Support pages for CIE (Cambridge International) A level students and teachers.</b></font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2"><a href="atommze="2">I have removed the Google search box because it was giving problems. Follow this link to find out how you can still search Chemguide using keywords.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2"><b>Edexcel Chemistry book</b></font></td><td><font color="#ff0000" face="Helvetica, Arial" size="2"><b>Support pages for my Edexcel International GCSE Chemistry book. This will soon be retitled as Edexcel International GCSE Chemistry, Edexcel Certificate inÌÌÌÌÌÌÌÌÌenu.html#top">Atomic Structure and Bonding</a></font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Covers basic atomic properties (electronic structures, ionisation energies, electron affinities, atomic and ionic radii, and the atomic hydrogen emission spectrum), bonding (including intermolecular bonding) and structures (ionic, molecular, giant covalent and metallic).</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Inorganic Chemistry</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Includes essential ideas about redox reactions, and covers the trends in Period 3 and Groups 1, 2, 4 and 7 of the Periodic Table. Plus: lengthy sections on the chemistry of some important complex ions, and of common transition metals. Extraction and uses of aluminium, copper, iron, titanium and tungsten.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" sÌÌÌÌÌÌÌÌèize="2">Physical Chemistry</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Covers simple kinetic theory, ideal and real gases, chemical energetics, rates of reaction including catalysis, an introduction to chemical equilibria, redox equilibria, acid-base equilibria (pH, buffer solutions, indicators, etc), solubility products, and phase equilibria (including Raoult's Law and the use of various phase diagetica, Arial" size="2">Inorganic Chemistry</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Includes essential ideas about redox reactions, and covers the trends in Period 3 and Groups 1, 2, 4 and 7 of the Periodic Table. Plus: lengthy sections on the chemistry of some important complex ions, and of common transition metals. Extraction and uses of aluminium, copper, iron, titanium and tungsten.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" sÌÌÌÌÌÌÌÌÌrams).</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Instrumental analysis</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Explains how you can analyse substances using machines - mass spectrometry, infra-red spectroscopy, NMR, UV-visible absorption spectrometry and chromatography.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Basic Organic Chemistry</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Includes help on bonding, naming and isomerism, and a discussion of organic acids and bases.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Properties of organic compounds</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Covers the physical and chemical properties of compounds on UK A ÌÌÌÌÌÌÌÌèlevel chemistry syllabuses, and includes a limited amount of biochemistry.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Organic Reaction Mechanisms</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Covers all the mechanisms required by the current UK A level chemistry syllabuses.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">About this site</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Includes a contact address if you have found any difficulties with the site.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Questions and comments</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">A selection of questions that I have been asked lots of times about Chemguide together with a few general commenÌÌÌÌÌÌÌÌèts. There are also a number of chemistry questions that I have been asked and which I haven't been able to find good answers for!</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Chemistry Calculations</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">A description of the author's book on calculations at UK A level chemistry standard.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Textbook suggestions</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">Suggestions for textbooks and revision guides covering the UK AS and A level chemistry syllabuses, with links to Amazon.co.uk if you want to follow them up.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Download syllabuses</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">For UK students and international students using UK exams (e.g. Cambridge International). Download a copy of your current syllabus from your examiners.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Links</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">A random collection of links to sites that I have found interesting or useful. You will find it is a fairly quirky collection - that's deliberate.</font></td></tr>
</table>
<pre>
</pre>
<hr />
<p><font color="#000000" size="2" face="Helvetica, Arial"> © Jim Clark 2009 (last modified September 2013)</font></p>
</td>
</tr>
</table></center>
</BODY>
</HTML>
tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Download syllabuses</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">For UK students and international students using UK exams (e.g. Cambridge International). Download a copy of your current syllabus from your examiners.</font></td></tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2">Links</font></td><td><font color="#000000" face="Helvetica, Arial" size="2">A random collection of links to sites that I have found interesting or useful. You will find it is a fairly quirky collection - that's deliberate.</font></td></tr>
</table>
<pre>
</pre>
<hr />
<p><font color="#000000" size="2" face="Helvetica, Arial"> © Jim Clark 2009 (last modified September 2013)</font></p>
</td>
</tr>
</table></center>
</BODY>
</HTML>
tr>
<tr valign="top"><td><font color="#000000" face="Helvetica, Arial" size="2"><a href="syllabushave been asked lots of times about Chemguide together with a few general commenÌÌÌÌÌÌÌÌ
I'm using Visual Studio 2013. Here's my stdafx.h file:
// stdafx.h : include file for standard system include files,
// or project specific include files that are used frequently, but
// are changed infrequently
//
#pragma once
#define _WINSOCK_DEPRECATED_NO_WARNINGS
//#define _CRT_SECURE_NO_WARNINGS
#include "targetver.h"
#include <stdio.h>
#include <tchar.h>
// TODO: reference additional headers your program requires here
The problem is that you treat the data you read as strings, but you seem to forget that C-style strings in C++ are terminated by the special character '\0'.
So you need to read one character less than the buffer size, and terminate the buffer you read as a string by adding the terminator character at the end:
if (i >= 0)
buffer[i] = '\0';
The reason you're getting gibberish is because when you append the buffer to the string server_reply, the += operator function looks for this terminator to find the end of the string to append, if the terminator the += operator function will just continue until it finds a byte corresponding to the terminator character, which might even be beyond the limits of buffer. Not terminating a string leads to undefined behavior.
Also, you don't check for errors when receiving, what do you think will happen if recv returns SOCKET_ERROR (which is not equal to zero)? You will end up with an infinite loop.
I have the following HTML codes and would like to use Google Script to obtain 39.85 and 2.49*.
<table>
<tr>
<td bgcolor="#CCCCCC">
<font size="4" color="#000000" >Malaysia RM </font>
</td>
<td bgcolor="#CCCCCC">
<font size="4" color="#000000" >39.85</font>
</td>
<td bgcolor="#CCCCCC" >
<font size="4" color="#000000" >2.49*</font>
</td>
</tr>
</table>
I uses following Google Scripts but got back 0.
var page = UrlFetchApp.fetch('http://.......');
var pageHTML = page.getContentText();
var xml = Xml.parse(pageHTML, true);
var mainTable = xml.getElement("body").getElement("table").getElement("tr");
var buy = mainTable.getElements("td")[1].getText();
var sell = mainTable.getElements("td")[2].getText();
I found the desired texts are bounded by font inside each td. Is there any API to get around this font section and retrieve the text.
I use this general purpose function from this answer (Corey G) to get Text from html and it works flawlessly.
function getTextFromHtml(html) {
return getTextFromNode(Xml.parse(html, true).getElement());
}
function getTextFromNode(x) {
switch(x.toString()) {
case 'XmlText': return x.toXmlString();
case 'XmlElement': return x.getNodes().map(getTextFromNode).join('');
default: return '';
}
}
function test(){
var xx='<table><tr><td bgcolor="#CCCCCC"><font size="4" color="#000000" >Malaysia RM </font></td><td bgcolor="#CCCCCC"><font size="4" color="#000000" >39.85</font></td><td bgcolor="#CCCCCC" ><font size="4" color="#000000" >2.49*</font></td></tr></table>'
Logger.log(getTextFromHtml(xx))
}
// Log result : [13-07-16 08:50:52:942 CEST] Malaysia RM 39.852.49*
From there a few simple string operations would allow you to retrieve the values you want (using dots as markers for example).