NameError: name 'status_code' is not defined while parsing access.log - json

Question

Good afternoon, while testing the code for parsing access.log, the following error occurred:
Traceback (most recent call last):
File "logsscript_3.py", line 31, in
dict_ip[ip][status_code] += 1
NameError: name 'status_code' is not defined
I need to output top 10 requests with code 400 to a json file
The code is like this:
import argparse
import json
import re
from collections import defaultdict
parser = argparse.ArgumentParser(description='log parser')
parser.add_argument('-f', dest='logfile', action='store', default='access.log')
args = parser.parse_args()
regul_ip = (r"^(?P<ips>.*?)")
regul_statuscode = (r"\s(?P<status_code>400)\s")
dict_ip = defaultdict(lambda: {"400": 0})
with open(args.logfile) as file:
for index, line in enumerate(file.readlines()):
try:
ip = re.search(regul_ip, line).group()
status_code = re.search(regul_statuscode, line).groups()[0]
except AttributeError:
pass
dict_ip[ip][status_code] += 1
print(json.dumps(dict_ip, indent=4))
with open("final_log.json", "w") as jsonfile:
json.dump(dict_ip, jsonfile, indent=5)
An example of a line from access.log:
213.137.244.2 - - [13/Dec/2015:17:30:13 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" 7717

Answer 1

Following up on the comment (included below for completeness) where I explain why you see the error, below I explain some ways to fix the code.
Expanding on #khelwood point: the example line (and likely many more in your log) is not a 400 code line. Your regex includes 400, thus it will not match, and the entire status_code = ... line will fail with an AttributeError: 'NoneType' object has no attribute 'groups' for all non-400 lines. Ignoring the exception results in a NameError in the dict_ip[ip]... line b/c status_code was not assigned a value.
First, you can use one regex to parse the access logs.
>>> import re
>>>
>>> line = '213.137.244.2 - - [13/Dec/2015:17:30:13 +0100] "GET /administrator/ HTTP/1.1" 200 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" 7717'
>>> p = r'(\S+) (\S+) (\S+) \[(.*?)\] "(\S+) (\S+) (\S+)" (\S+) (\S+) "(\S+)" "(.*?)" (\S+)'
>>> pat = re.compile(p)
>>> m = pat.match(line)
>>> m.groups()
('213.137.244.2', '-', '-', '13/Dec/2015:17:30:13 +0100', 'GET', '/administrator/', 'HTTP/1.1', '200', '4263', '-', 'Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0', '7717')
>>> m.group(1)
'213.137.244.2'
>>> m.group(2)
'-'
...
The snippet above shows you how to fetch and access the various fields from your log, as I observed in your other recent questions that you need this.
You can slightly modify the above as shown below (since you care only about log lines with a 400 and need only the IP address). Note that this is not the only way to write the regex, it's simply one way that can be easily derived from the one above. Note also that for illustration purposes I changed 200 to 400.
>>> line = '213.137.244.2 - - [13/Dec/2015:17:30:13 +0100] "GET /administrator/ HTTP/1.1" 400 4263 "-" "Mozilla/5.0 (Windows NT 6.0; rv:34.0) Gecko/20100101 Firefox/34.0" 7717'
>>> p = r'(\S+) \S+ \S+ \[.*?\] "\S+ \S+ \S+" 400 \S+ "\S+" ".*?" \S+'
>>> pat = re.compile(p)
>>> m = pat.match(line)
>>> m.group(1)
'213.137.244.2'
So, reading your log, counting 400s per IP address, and saving the 10 IP addresses with most 400s in a json file:
>>> from collections import Counter
>>> import json
>>> import re
>>>
>>> p = r'(\S+) \S+ \S+ \[.*?\] "\S+ \S+ \S+" 400 \S+ "\S+" ".*?" \S+'
>>> pat = re.compile(p)
>>> dict_ips_400 = Counter()
>>>
>>> with open("input_log.text") as f:
>>> for line in f: # see Note 1
>>> m = pat.match(line)
>>> if m: # check if there is a match
>>> ip = m.group(1)
>>> dict_ips_400[ip] += 1
>>>
>>> with open("final_log.json", "w") as jsonfile:
... json.dump(dict_ips_400.most_common(10), jsonfile, indent=5)
...
Notes:
you may want to check the differences of using f.readlines() vs processing a file line by line as above (if you are working with large files)
you could modify the above to a. use named groups (see re's docs), as you attempted to do in your code and/or b. capture and store more fields, say, IP addresses and status code pair counts