How to better deal with regex capture group between HTML tags? - html

Question

I'm trying to capture this content inside the html tag document in the string below. The result yields the desired match, but also a weird entry "t", the last letter before the close tag.
I'm pretty new to regex and I wonder what is going on? What should I read up about?
PS: If I remove the () brackets around the pattern, only 't' is captured. I'm not sure I can see what difference the bracket (i.e. defining a capture group) make in this case.
example = '''ABCDE<DOCUMENT>
Lorem ipsum
dolor sit amet</DOCUMENT>
EFGHIJK.'''
re.findall(r'(<DOCUMENT>(.|\s)*<\/DOCUMENT>)', example)
Outputs:
[('<DOCUMENT>\nLorem ipsum\ndolor sit amet</DOCUMENT>', 't')]

Answer 1

Try using the re.DOTALL flag instead of using \s to capture whitespaces:
re.findall(r'(<DOCUMENT>.*<\/DOCUMENT>)', example, flags = re.DOTALL)
Explaining the issue
re.findall documentation states that:
If one or more groups are present in the pattern, return a list of groups; this will be a list of tuples if the pattern has more than one group
You have two capturing groups (defined by the parenthesis) in your regex:
over all the pattern, defined by the first and last parenthesis
over the .|\s pattern
That's why the return is a list of a tuple with two elements: \nLorem ipsum\ndolor sit amet and t.
When you use the * outside the capturing group, you are actually matching it multiple times. The last time the group matches, is the last t of "amet" in the input string, thus findall returns it as the value of the capturing group.

Answer 2

Here, we can use this expression,
<DOCUMENT>(.*?)<\/DOCUMENT>
Please see this demo for explanation.
with s flag, or any of these expressions:
<DOCUMENT>([\s\S]*?)<\/DOCUMENT>
<DOCUMENT>([\d\D]*?)<\/DOCUMENT>
<DOCUMENT>([\w\W]*?)<\/DOCUMENT>
with m flag, and our problem would be likely solved.
Please see this demo for explanation.
Test
import re
regex = r"<DOCUMENT>([\s\S]*?)<\/DOCUMENT>"
test_str = ("ABCDE<DOCUMENT>\n"
"Lorem ipsum\n\n\n\n"
"dolor sit amet</DOCUMENT>\n"
"EFGHIJK.")
matches = re.finditer(regex, test_str, re.MULTILINE)
for matchNum, match in enumerate(matches, start=1):
print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group()))
for groupNum in range(0, len(match.groups())):
groupNum = groupNum + 1
print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))