Handling malformed HTML with Go's net/html tokenizer? - html

Question

I’ve found that the html.NewTokenizer() doesn’t auto-fix some things. So it’s possible that you can end up with a stray closing tag (html.EndTagToken). So <div></p></div> would be html.StartTagToken, html.EndTagToken, html.EndTagToken.
Is there a recommended solution for handling ignoring/removing/fixing these tags?
My first guess would be manually keeping a []atom.Atom slice and push/pop to the list as you start/end each tag (after comparing the tag to make sure you don’t get an unexpected end tag).
Here is some code to demonstrate the problem:
var err error
htm := `<div><div><p></p></p></div>`
tokenizer := html.NewTokenizer(strings.NewReader(htm))
for {
if tokenizer.Next() == html.ErrorToken {
err = tokenizer.Err()
if err == io.EOF {
err = nil
}
return
}
token := tokenizer.Token()
switch token.Type {
case html.DoctypeToken:
continue
case html.CommentToken:
continue
case html.SelfClosingTagToken:
fmt.Println(token.Data)
continue
case html.StartTagToken:
fmt.Printf("<%s>\n", token.Data)
case html.EndTagToken:
fmt.Printf("</%s>\n", token.Data)
case html.TextToken:
continue
default:
continue
}
}
Output:
<div>
<div>
<p>
</p>
</p>
</div>

Answer 1

FWIW, it seems that net/html can fix such issues when you use its Parse method. Here's an example adapted from another SO answer, using your malformed HTML snippet:
package main
import (
"bytes"
"fmt"
"log"
"strings"
"golang.org/x/net/html"
)
func main() {
brokenHtml := `<div><div><p></p></p></div>`
reader := strings.NewReader(brokenHtml)
root, err := html.Parse(reader)
if err != nil {
log.Fatal(err)
}
var b bytes.Buffer
html.Render(&b, root)
fixedHtml := b.String()
fmt.Println(fixedHtml)
}