Login to https website with Python with request - html

Question

I'd like to access some content from https://bato.to/ that requires me to login first. Their login page is: https://bato.to/forums/index.php?app=core&module=global&section=login
I've opened chrome's web developer tools to inspect the POST that's sent when I click login. The 'Form Data' inside the POST is:
auth_key:880ea6a14ea49e853634fbdc5015a024
referer:https://bato.to/forums/
ips_username:startwinkling
ips_password:password1
rememberMe:1
So I've tried to implement this with the code:
Code so far
from requests import session
import re
AUTH_KEY = re.compile(r"<input type='hidden' name='auth_key' value='(.*?)' \/>")
payload = {
'ips_username': 'startwinkling',
'ips_password': 'password1',
'rememberMe' : '1',
'referer' : 'https://bato.to/forums/'
}
with session() as c:
login_url = 'https://bato.to/forums/index.php?app=core&module=global&section=login'
page = c.get(login_url)
auth_key = AUTH_KEY.search(page.text).group(1)
payload['auth_key'] = auth_key
print("auth_key: %s" % auth_key)
page = c.post(login_url, data=payload)
page = c.get('https://bato.to/reader#4b57865eb3a9a9a6')
print(page.text)
I believe I'm grabbing and passing in the auth_key properly since the code outputs:
auth_key: 880ea6a14ea49e853634fbdc5015a024
But the HTML that's printed out indicate that I haven't been able to successfully log in. What am I missing here?

Answer 1

The URL you use for POST is not correct.
The correct one should be https://bato.to/forums/index.php?app=core&module=global&section=login&do=process, it's not the same as login landing page, notice the extra do=process part.
Codes:
from requests import session
import re
AUTH_KEY = re.compile(r"<input type='hidden' name='auth_key' value='(.*?)' \/>")
payload = {
'ips_username': 'startwinkling',
'ips_password': 'password1',
'rememberMe' : '1',
'referer' : 'https://bato.to/forums/'
}
with session() as c:
login_url = 'https://bato.to/forums/index.php?app=core&module=global&section=login'
page = c.get(login_url)
auth_key = AUTH_KEY.search(page.text).group(1)
payload['auth_key'] = auth_key
print("auth_key: %s" % auth_key)
page = c.post(login_url + '&do=process', data=payload)
page = c.get('https://bato.to/reader#4b57865eb3a9a9a6')
print(page.text)
P.S. I would suggest you to add some headers(not use default headers) as well, you might not want to appear as User-Agent: python-requests/1.2.3 CPython/2.7.3 Windows/7 on their analytics, also in case they set some limits on certain pages for "non-browser" visit.