Extract plain text from an html file in C [duplicate] - html

This question already exists:
Using Regex in C [closed]
Closed 9 years ago.
I am really desperate. I need to extract all html elements including html tags. I want to retain just plain text. I am required to do this in C. I am discouraged to use Regex. If I use string functions, it just removes delimiters , not the string inside. I need to create a program which extracts plain text from an html file. Any guide would be appreciated on how to do so. Thanks!

Here's a starting point for you:
void remove_html(char* str) {
char* html_str = str;
while(*str) {
if(*html_str == '<')
while(*html_str && *html_str++ != '>');
*str++ = *html_str++;
}
}
int main() {
char foo[] = "hello <p>friends<b>!</b></p>";
remove_html(foo);
puts(foo);
}
It only strips the angular syntax - doesn't do any parsing. Also, it doesn't convert escape characters.

If you open up a html file in notepad, you'll find it is plain text (no images or anything).
All tags start with < and end with >, everything else is text. In this way, you can read through the file only once, excluding the characters that appear between < > symbols.
Pseudocode:
bool intag=false;
for (i=0;i<filesize;i++) {
char c = readchar();
if (c=='<') intag=true;
if (!intag) writechar(c);
if (c=='>') intag=false;
This logic should work for most cases, though you may have to do some more work to deal with indented text and possibly any javascript on the page.

Related

Appending long snippets of html in jquery [duplicate]

I have the following code in Ruby. I want to convert this code into JavaScript. What is the equivalent code in JS?
text = <<"HERE"
This
Is
A
Multiline
String
HERE
Update:
ECMAScript 6 (ES6) introduces a new type of literal, namely template literals. They have many features, variable interpolation among others, but most importantly for this question, they can be multiline.
A template literal is delimited by backticks:
var html = `
<div>
<span>Some HTML here</span>
</div>
`;
(Note: I'm not advocating to use HTML in strings)
Browser support is OK, but you can use transpilers to be more compatible.
Original ES5 answer:
Javascript doesn't have a here-document syntax. You can escape the literal newline, however, which comes close:
"foo \
bar"
ES6 Update:
As the first answer mentions, with ES6/Babel, you can now create multi-line strings simply by using backticks:
const htmlString = `Say hello to
multi-line
strings!`;
Interpolating variables is a popular new feature that comes with back-tick delimited strings:
const htmlString = `${user.name} liked your post about strings`;
This just transpiles down to concatenation:
user.name + ' liked your post about strings'
Original ES5 answer:
Google's JavaScript style guide recommends to use string concatenation instead of escaping newlines:
Do not do this:
var myString = 'A rather long string of English text, an error message \
actually that just keeps going and going -- an error \
message to make the Energizer bunny blush (right through \
those Schwarzenegger shades)! Where was I? Oh yes, \
you\'ve got an error and all the extraneous whitespace is \
just gravy. Have a nice day.';
The whitespace at the beginning of each line can't be safely stripped at compile time; whitespace after the slash will result in tricky errors; and while most script engines support this, it is not part of ECMAScript.
Use string concatenation instead:
var myString = 'A rather long string of English text, an error message ' +
'actually that just keeps going and going -- an error ' +
'message to make the Energizer bunny blush (right through ' +
'those Schwarzenegger shades)! Where was I? Oh yes, ' +
'you\'ve got an error and all the extraneous whitespace is ' +
'just gravy. Have a nice day.';
the pattern text = <<"HERE" This Is A Multiline String HERE is not available in js (I remember using it much in my good old Perl days).
To keep oversight with complex or long multiline strings I sometimes use an array pattern:
var myString =
['<div id="someId">',
'some content<br />',
'someRefTxt',
'</div>'
].join('\n');
or the pattern anonymous already showed (escape newline), which can be an ugly block in your code:
var myString =
'<div id="someId"> \
some content<br /> \
someRefTxt \
</div>';
Here's another weird but working 'trick'1:
var myString = (function () {/*
<div id="someId">
some content<br />
someRefTxt
</div>
*/}).toString().match(/[^]*\/\*([^]*)\*\/\}$/)[1];
external edit: jsfiddle
ES20xx supports spanning strings over multiple lines using template strings:
let str = `This is a text
with multiple lines.
Escapes are interpreted,
\n is a newline.`;
let str = String.raw`This is a text
with multiple lines.
Escapes are not interpreted,
\n is not a newline.`;
1 Note: this will be lost after minifying/obfuscating your code
You can have multiline strings in pure JavaScript.
This method is based on the serialization of functions, which is defined to be implementation-dependent. It does work in the most browsers (see below), but there's no guarantee that it will still work in the future, so do not rely on it.
Using the following function:
function hereDoc(f) {
return f.toString().
replace(/^[^\/]+\/\*!?/, '').
replace(/\*\/[^\/]+$/, '');
}
You can have here-documents like this:
var tennysonQuote = hereDoc(function() {/*!
Theirs not to make reply,
Theirs not to reason why,
Theirs but to do and die
*/});
The method has successfully been tested in the following browsers (not mentioned = not tested):
IE 4 - 10
Opera 9.50 - 12 (not in 9-)
Safari 4 - 6 (not in 3-)
Chrome 1 - 45
Firefox 17 - 21 (not in 16-)
Rekonq 0.7.0 - 0.8.0
Not supported in Konqueror 4.7.4
Be careful with your minifier, though. It tends to remove comments. For the YUI compressor, a comment starting with /*! (like the one I used) will be preserved.
I think a real solution would be to use CoffeeScript.
ES6 UPDATE: You could use backtick instead of creating a function with a comment and running toString on the comment. The regex would need to be updated to only strip spaces. You could also have a string prototype method for doing this:
let foo = `
bar loves cake
baz loves beer
beer loves people
`.removeIndentation()
Someone should write this .removeIndentation string method... ;)
You can do this...
var string = 'This is\n' +
'a multiline\n' +
'string';
I came up with this very jimmy rigged method of a multi lined string. Since converting a function into a string also returns any comments inside the function you can use the comments as your string using a multilined comment /**/. You just have to trim off the ends and you have your string.
var myString = function(){/*
This is some
awesome multi-lined
string using a comment
inside a function
returned as a string.
Enjoy the jimmy rigged code.
*/}.toString().slice(14,-3)
alert(myString)
I'm surprised I didn't see this, because it works everywhere I've tested it and is very useful for e.g. templates:
<script type="bogus" id="multi">
My
multiline
string
</script>
<script>
alert($('#multi').html());
</script>
Does anybody know of an environment where there is HTML but it doesn't work?
I solved this by outputting a div, making it hidden, and calling the div id by jQuery when I needed it.
e.g.
<div id="UniqueID" style="display:none;">
Strings
On
Multiple
Lines
Here
</div>
Then when I need to get the string, I just use the following jQuery:
$('#UniqueID').html();
Which returns my text on multiple lines. If I call
alert($('#UniqueID').html());
I get:
There are multiple ways to achieve this
1. Slash concatenation
var MultiLine= '1\
2\
3\
4\
5\
6\
7\
8\
9';
2. regular concatenation
var MultiLine = '1'
+'2'
+'3'
+'4'
+'5';
3. Array Join concatenation
var MultiLine = [
'1',
'2',
'3',
'4',
'5'
].join('');
Performance wise, Slash concatenation (first one) is the fastest.
Refer this test case for more details regarding the performance
Update:
With the ES2015, we can take advantage of its Template strings feature. With it, we just need to use back-ticks for creating multi line strings
Example:
`<h1>{{title}}</h1>
<h2>{{hero.name}} details!</h2>
<div><label>id: </label>{{hero.id}}</div>
<div><label>name: </label>{{hero.name}}</div>
`
Using script tags:
add a <script>...</script> block containing your multiline text into head tag;
get your multiline text as is... (watch out for text encoding: UTF-8, ASCII)
<script>
// pure javascript
var text = document.getElementById("mySoapMessage").innerHTML ;
// using JQuery's document ready for safety
$(document).ready(function() {
var text = $("#mySoapMessage").html();
});
</script>
<script id="mySoapMessage" type="text/plain">
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:typ="...">
<soapenv:Header/>
<soapenv:Body>
<typ:getConvocadosElement>
...
</typ:getConvocadosElement>
</soapenv:Body>
</soapenv:Envelope>
<!-- this comment will be present on your string -->
//uh-oh, javascript comments... SOAP request will fail
</script>
I like this syntax and indendation:
string = 'my long string...\n'
+ 'continue here\n'
+ 'and here.';
(but actually can't be considered as multiline string)
Downvoters: This code is supplied for information only.
This has been tested in Fx 19 and Chrome 24 on Mac
DEMO
var new_comment; /*<<<EOF
<li class="photobooth-comment">
<span class="username">
You:
</span>
<span class="comment-text">
$text
</span>
#<span class="comment-time">
2d
</span> ago
</li>
EOF*/
// note the script tag here is hardcoded as the FIRST tag
new_comment=document.currentScript.innerHTML.split("EOF")[1];
document.querySelector("ul").innerHTML=new_comment.replace('$text','This is a dynamically created text');
<ul></ul>
A simple way to print multiline strings in JavaScript is by using template literals(template strings) denoted by backticks (` `). you can also use variables inside a template string-like (` name is ${value} `)
You can also
const value = `multiline`
const text = `This is a
${value}
string in js`;
console.log(text);
There's this library that makes it beautiful:
https://github.com/sindresorhus/multiline
Before
var str = '' +
'<!doctype html>' +
'<html>' +
' <body>' +
' <h1>❤ unicorns</h1>' +
' </body>' +
'</html>' +
'';
After
var str = multiline(function(){/*
<!doctype html>
<html>
<body>
<h1>❤ unicorns</h1>
</body>
</html>
*/});
Found a lot of over engineered answers here.
The two best answers in my opinion were:
1:
let str = `Multiline string.
foo.
bar.`
which eventually logs:
Multiline string.
foo.
bar.
2:
let str = `Multiline string.
foo.
bar.`
That logs it correctly but it's ugly in the script file if str is nested inside functions / objects etc...:
Multiline string.
foo.
bar.
My really simple answer with regex which logs the str correctly:
let str = `Multiline string.
foo.
bar.`.replace(/\n +/g, '\n');
Please note that it is not the perfect solution but it works if you are sure that after the new line (\n) at least one space will come (+ means at least one occurrence). It also will work with * (zero or more).
You can be more explicit and use {n,} which means at least n occurrences.
The equivalent in javascript is:
var text = `
This
Is
A
Multiline
String
`;
Here's the specification. See browser support at the bottom of this page. Here are some examples too.
This works in IE, Safari, Chrome and Firefox:
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js"></script>
<div class="crazy_idea" thorn_in_my_side='<table border="0">
<tr>
<td ><span class="mlayouttablecellsdynamic">PACKAGE price $65.00</span></td>
</tr>
</table>'></div>
<script type="text/javascript">
alert($(".crazy_idea").attr("thorn_in_my_side"));
</script>
to sum up, I have tried 2 approaches listed here in user javascript programming (Opera 11.01):
this one didn't work: Creating multiline strings in JavaScript
this worked fairly well, I have also figured out how to make it look good in Notepad++ source view: Creating multiline strings in JavaScript
So I recommend the working approach for Opera user JS users. Unlike what the author was saying:
It doesn't work on firefox or opera; only on IE, chrome and safari.
It DOES work in Opera 11. At least in user JS scripts. Too bad I can't comment on individual answers or upvote the answer, I'd do it immediately. If possible, someone with higher privileges please do it for me.
Exact
Ruby produce: "This\nIs\nA\nMultiline\nString\n" - below JS produce exact same string
text = `This
Is
A
Multiline
String
`
// TEST
console.log(JSON.stringify(text));
console.log(text);
This is improvement to Lonnie Best answer because new-line characters in his answer are not exactly the same positions as in ruby output
My extension to https://stackoverflow.com/a/15558082/80404.
It expects comment in a form /*! any multiline comment */ where symbol ! is used to prevent removing by minification (at least for YUI compressor)
Function.prototype.extractComment = function() {
var startComment = "/*!";
var endComment = "*/";
var str = this.toString();
var start = str.indexOf(startComment);
var end = str.lastIndexOf(endComment);
return str.slice(start + startComment.length, -(str.length - end));
};
Example:
var tmpl = function() { /*!
<div class="navbar-collapse collapse">
<ul class="nav navbar-nav">
</ul>
</div>
*/}.extractComment();
Updated for 2015: it's six years later now: most people use a module loader, and the main module systems each have ways of loading templates. It's not inline, but the most common type of multiline string are templates, and templates should generally be kept out of JS anyway.
require.js: 'require text'.
Using require.js 'text' plugin, with a multiline template in template.html
var template = require('text!template.html')
NPM/browserify: the 'brfs' module
Browserify uses a 'brfs' module to load text files. This will actually build your template into your bundled HTML.
var fs = require("fs");
var template = fs.readFileSync(template.html', 'utf8');
Easy.
If you're willing to use the escaped newlines, they can be used nicely. It looks like a document with a page border.
Easiest way to make multiline strings in Javascrips is with the use of backticks ( `` ). This allows you to create multiline strings in which you can insert variables with ${variableName}.
Example:
let name = 'Willem';
let age = 26;
let multilineString = `
my name is: ${name}
my age is: ${age}
`;
console.log(multilineString);
compatibility :
It was introduces in ES6//es2015
It is now natively supported by all major browser vendors (except internet explorer)
Check exact compatibility in Mozilla docs here
The ES6 way of doing it would be by using template literals:
const str = `This
is
a
multiline text`;
console.log(str);
More reference here
You can use TypeScript (JavaScript SuperSet), it supports multiline strings, and transpiles back down to pure JavaScript without overhead:
var templates = {
myString: `this is
a multiline
string`
}
alert(templates.myString);
If you'd want to accomplish the same with plain JavaScript:
var templates =
{
myString: function(){/*
This is some
awesome multi-lined
string using a comment
inside a function
returned as a string.
Enjoy the jimmy rigged code.
*/}.toString().slice(14,-3)
}
alert(templates.myString)
Note that the iPad/Safari does not support 'functionName.toString()'
If you have a lot of legacy code, you can also use the plain JavaScript variant in TypeScript (for cleanup purposes):
interface externTemplates
{
myString:string;
}
declare var templates:externTemplates;
alert(templates.myString)
and you can use the multiline-string object from the plain JavaScript variant, where you put the templates into another file (which you can merge in the bundle).
You can try TypeScript at
http://www.typescriptlang.org/Playground
ES6 allows you to use a backtick to specify a string on multiple lines. It's called a Template Literal. Like this:
var multilineString = `One line of text
second line of text
third line of text
fourth line of text`;
Using the backtick works in NodeJS, and it's supported by Chrome, Firefox, Edge, Safari, and Opera.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals
Also do note that, when extending string over multiple lines using forward backslash at end of each line, any extra characters (mostly spaces, tabs and comments added by mistake) after forward backslash will cause unexpected character error, which i took an hour to find out
var string = "line1\ // comment, space or tabs here raise error
line2";
Please for the love of the internet use string concatenation and opt not to use ES6 solutions for this. ES6 is NOT supported all across the board, much like CSS3 and certain browsers being slow to adapt to the CSS3 movement. Use plain ol' JavaScript, your end users will thank you.
Example:
var str = "This world is neither flat nor round. "+
"Once was lost will be found";
You can use tagged templates to make sure you get the desired output.
For example:
// Merging multiple whitespaces and trimming the output
const t = (strings) => { return strings.map((s) => s.replace(/\s+/g, ' ')).join("").trim() }
console.log(t`
This
Is
A
Multiline
String
`);
// Output: 'This Is A Multiline String'
// Similar but keeping whitespaces:
const tW = (strings) => { return strings.map((s) => s.replace(/\s+/g, '\n')).join("").trim() }
console.log(tW`
This
Is
A
Multiline
String
`);
// Output: 'This\nIs\nA\nMultiline\nString'
Multiline string with variables
var x = 1
string = string + `<label class="container">
<p>${x}</p>
</label>`;

Regex only captures the last occurrence of a match in html format

I've been learning regex and for that I have been working on Hackerrank problems. I came across a problem where I am asked to remove html format and only keep whatever is inside an anchor tag's reference (the value of the href part), and the text inside the tag, then present this separated by a comma.
I came up with the following code to extract such information:
public static void main(String[] args) {
Scanner s = new Scanner(System.in);
int n = s.nextInt();
s.nextLine();
for (int i = 0; i < n; i++) {
String line = s.nextLine();
Pattern p = Pattern.compile("(.*)<a href=\"([^\"]+)\"([^<>]*)>(<\\w+>)*([^<>]+)</a>(</\\w+>)*");
Matcher m = p.matcher(line);
while (m.find()) {
System.out.println(m.group(2).trim() + "," + m.group(5).trim());
}
}
}
This code, when presented with cases such as <p>text</p> Passes and outputs folder/page,text
But if the input has multiple <a> tags, it will only grab the last occurrence of it and output that, instead of outputting all possible matches for that single input. Why is this happening? Please don't feel obliged to answer my question fully if you think I can answer it myself with just a hint. Thank you for any answers in advance

Convert QString to text with substitutes for HTML special characters (e.g. tags)

The user will be able to put in some text into a QLineEdit in a Qt environment. However, these input texts can contain HTML special characters. My aim is to convert this text by replacing all HTML special character occurences with substitutes.
A similar case is found in PHP with the htmlspecialchars() function http://php.net/manual/en/function.htmlspecialchars.php.
The main reason I want to do this is because I want to display the user input in a richtext QTextEdit and I don't want the user to be able to change HTML and I wish to be able to use HTML special characters without too much hassle.
How can this be achieved?
The easiest way I know, is to use QTextEdit::toHtml:
QString convert();
{
QString s = lineEdit->text();
QTextEdit textEdit;
textEdit.setPlainText(s);
QString ret = textEdit.toHtml();
int firstClosingTag = ret.indexOf("</p></body></html>");
int lastOpeningTag = ret.lastIndexOf(">", firstClosingTag);
return ret.mid(lastOpeningTag + 1, firstClosingTag - lastOpeningTag - 1);
}
There are also two functions, which you could find useful:
Qt::convertFromPlainText() and Qt::escape()
In Qt5, it's QString::toHtmlEscaped, e.g.:
QString a = "Hello, <span class=\"name\">Bear</span>!";
// a will contain: Hello, <span class="name">Bear</span>!
QString b = a.toHtmlEscaped();
// b will contain: Hello, <span class="name">Bear</span>!
This is direct equivalent of the htmlspecialchars in PHP. It replaces the Qt::escape function (mentioned by Amartel), which does the same thing but is now obsolete.
The Qt::convertFromPlainText function (also mentioned by Amartel) still exists in Qt 5, but it does more than PHP's htmlspecialchars. Not only it replaces < with <, > with >, & with &, " with " but also does additional handling of whitespace characters (space, tab, line feed, etc) to make the generated HTML look visually similarly to the original plain text. Particularly, it may put <p>…</p>/<br> for linefeeds, non-breaking spaces for spaces and multiple non-breaking spaces for tabs. I.e. this function is not just htmlspecialchars, it's even more comprehensive than nl2br(htmlspecialchars($s)) combination.
Note that unlike the PHP's htmlspecialchars with ENT_QUOTES, none of the Qt functions listed in this answer replace single quote (') with &apos;/'. So, for example, QString html = "<img alt='" + s.toHtmlEscaped() + "'>"; won't be safe, only QString html = "<img alt=\"" + s.toHtmlEscaped() + "\">"; will. (However, as < is replaced and ' has no special meaning outside <…>, something like QString html = "<b>" + s.toHtmlEscaped() + "</b>"; would also be safe.)

JSFL: convert text from a textfield to a HTML-format string

I've got a deceptively simple question: how can I get the text from a text field AND include the formatting? Going through the usual docs I found out it is possible to get the text only. It is also possible to get the text formatting, but this only works if the entire text field uses only one kind of formatting. I need the precise formatting so that I convert it to a string with html-tags.
Personally I need this so I can pass it to a custom-made text field component that uses HTML for formatting. But it could also be used to simply export the contents of any text field to any other format. This could be of interest to others out there, too.
Looking for a solution elsewhere I found this:
http://labs.thesedays.com/blog/2010/03/18/jsfl-rich-text/
Which seems to do the reverse of what I need, convert HTML to Flash Text. My own attempts to reverse this have not been successful thus far. Maybe someone else sees an easy way to reverse this that I’m missing? There might also be other solutions. One might be to get the EXACT data of the text field, which should include formatting tags of some kind(XML, when looking into the contents of the stored FLA file). Then remove/convert those tags. But I have no idea how to do this, if at all possible. Another option is to cycle through every character using start- and endIndex, and storing each formatting kind in an array. Then I could apply the formatting to each character. But this will result in excess tags. Especially for hyperlinks! So can anybody help me with this?
A bit late to the party but the following function takes a JSFL static text element as input and returns a HTML string (using the Flash-friendly <font> tag) based on the styles found it its TextRuns array. It's doing a bit of basic regex to clear up some tags and double spaces etc. and convert /r and /n to <br/> tags. It's probably not perfect but hopefully you can see what's going on easy enough to change or fix it.
function tfToHTML(p_tf)
{
var textRuns = p_tf.textRuns;
var html = "";
for ( var i=0; i<textRuns.length; i++ )
{
var textRun = textRuns[i];
var chars = textRun.characters;
chars = chars.replace(/\n/g,"<br/>");
chars = chars.replace(/\r/g,"<br/>");
chars = chars.replace(/ /g," ");
chars = chars.replace(/. <br\/>/g,".<br/>");
var attrs = textRun.textAttrs;
var font = attrs.face;
var size = attrs.size;
var bold = attrs.bold;
var italic = attrs.italic;
var colour = attrs.fillColor;
if ( bold )
{
chars = "<b>"+chars+"</b>";
}
if ( italic )
{
chars = "<i>"+chars+"</i>";
}
chars = "<font size=\""+size+"\" face=\""+font+"\" color=\""+colour+"\">"+chars+"</font>";
html += chars;
}
return html;
}

Converting tag contents to lipsum with regex [duplicate]

This question already has answers here:
RegEx match open tags except XHTML self-contained tags
(35 answers)
Closed 8 years ago.
I'm debranding a micro-site to use as a portfolio piece. It's built with static html, I need to replace the contents of every non-script tag with lipsum or even scrambled text - but it has to be the same number of characters as the current text to keep the formatting nice. Furthermore, I really would rather do this with GUI grep editor rather than writing a script because there may be a few tags I need to keep the contents of.
I used the regex \>([^$]+?)\< to find them (all the scripts start with $ so it skips the script tag) but I can't find any way to count the number of characters matched and replace with a corresponding number of lipsum or random characters.
Thanks for any help!
I was able to successfully do this, though I had to end up using a Java program. Turns out regex is fine cause I'm not parsing the whole thing, just a few parts. There are a few quirks but this got the job done.
public class Debrander {
public static void main(String[] args) {
// reads in html from StdIn
String htmlPage = StdIn.readAll();
//regex matches all content within non-script non-style tags
Pattern tagContentRegex = Pattern.compile("\\>(.*?)\\<(?!/script)(?!/style)");
Matcher myMatcher = tagContentRegex.matcher(htmlPage);
//different regex to check for whitespace
Pattern whiteRegex = Pattern.compile("[^\\s]");
StringBuffer sb = new StringBuffer();
LoremIpsum4J loremIpsum = new LoremIpsum4J();
loremIpsum.setStartWithLoremIpsum(false);
//loop through all matches
while(myMatcher.find()){
String tagContent = htmlPage.substring(myMatcher.start(1), myMatcher.end(1));
Matcher whiteMatcher = whiteRegex.matcher(tagContent);
//whiteMatcher makes sure there is a NON-WHITESPACE character in the string
if (whiteMatcher.find()){
Integer charCount = (myMatcher.end(1) - myMatcher.start(1));
String[] lipsum = loremIpsum.getBytes(charCount);
String replaceString = ">";
for (int i=0; i<lipsum.length; i++){
replaceString += lipsum[i];
}
replaceString += "<";
myMatcher.appendReplacement(sb, replaceString);
}
}
myMatcher.appendTail(sb);
StdOut.println(sb.toString());
}
}