Parse from log file in python - json

Question

I have a log file with arbitrary number of lines and json strings. All I need is to extract is one json data from the log file BUT ONLY AFTER '_____GP D_____'. I do not want any other lines or json data from the file.
This is how my input file looks
INFO:modules.gp.helpers.parameter_getter:_____GP D_____
{'from_time': '2017-07-12 19:57', 'to_time': '2017-07-12 20:57', 'consig_number': 'dup1', 'text': 'r155', 'mobile': None, 'email': None}
ERROR:modules.common.actionexception:ActionError: [{'other': 'your request already crossed threshold time'}]
{'from_time': '2016-07-12 16:57', 'to_time': '2016-07-12 22:57', 'consig_number': 'dup2', 'text': 'r15', 'mobile': None, 'email': None}
how do i find the json string only after '_____GP D_____'?

Answer 1

You can read your file line by line until you encounter _____GP D_____ at the end of the line, and when you do pick up just the next line:
found_json = None
with open("input.log", "r") as f: # open your log file
for line in f: # read it line by line
if line.rstrip()[-14:] == "_____GP D_____": # if a line ends with our string...
found_json = next(f).rstrip() # grab the next line
break # stop reading of the file, nothing more of interest
Then you can do with your found_json whatever you want, including parsing it, printing it, etc.
UPDATE - If you want to continuously 'follow' your log file (akin to the tail -f command) you can open it in read mode and keep the file handle open while reading it line by line with a reasonable delay added between reads (that's largely how tail -f does it, too) - then you can use the same procedure to discover when your desired line occurs and capture the next line to process, send to some other process or do whatever you plan to do with it. Something like:
import time
capture = False # a flag to use to signal the capture of the next line
found_lines = [] # a list to store our found lines, just as an example
with open("input.log", "r") as f: # open the file for reading...
while True: # loop indefinitely
line = f.readline() # grab a line from the file
if line != '': # if there is some content on the current line...
if capture: # capture the current line
found_lines.append(line.rstrip()) # store the found line
# instead, you can do whatever you want with the captured line
# i.e. to print it: print("Found: {}".format(line.rstrip()))
capture = False # reset the capture flag
elif line.rstrip()[-14:] == "_____GP D_____": # if it ends in '_____GP D_____'..
capture = True # signal that the next line should be captured
else: # an empty buffer encountered, most probably EOF...
time.sleep(1) # ... let's wait for a second before attempting to read again...

Answer 2

import json
from ast import literal_eval
KEY_STRING = '''_____GP D_____'''
text = """INFO:modules.gp.helpers.parameter_getter:_____GP D_____
{'from_time': '2017-07-12 19:57', 'to_time': '2017-07-12 20:57', 'consig_number': 'dup1', 'text': 'r155', 'mobile': None, 'email': None}
ERROR:modules.common.actionexception:ActionError: [{'other': 'your request already crossed threshold time'}]
{'from_time': '2016-07-12 16:57', 'to_time': '2016-07-12 22:57', 'consig_number': 'dup2', 'text': 'r15', 'mobile': None, 'email': None}"""
lines = text.split("\n") # load log text into a list.
# for loading from log would be more like
# with open("/var/log/syslog.log", 'r') as f:
# lines = f.readlines()
# set "gate" flag to False
flag = False
for loop in lines:
line = loop.strip()
if flag: # "gate" opened
# depends how's the dictionary streamed to log
# you could use json.loads(line), but if it is not sent to log with json.dumps than you have pythonic dictinary and use
# literal_eval to load that dictionary to a variable
# .. a
target_json = literal_eval(line)
print json.dumps(target_json, indent=4)
if KEY_STRING in line:
flag = True # KEY_STRING found open "gate"
else:
flag = False # close "gate"
~
Output:
{
"consig_number": "dup1",
"text": "r155",
"email": null,
"mobile": null,
"to_time": "2017-07-12 20:57",
"from_time": "2017-07-12 19:57"
}