web scrape save to specific json in python, bs4 - json

Question

I have the following Python code:
import requests
import json
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Mozilla/5.0 (Linux; Android 4.3; nl-nl; SAMSUNG GT-I9505 Build/JSS15J) AppleWebKit/537.36 (KHTML, like Gecko) Version/1.5 Chrome/28.0.1500.94 Mobile Safari/537.36'}
chapter = 0
while chapter < 3 :
url = 'http://www.komikgue.com/manga/one-piece/{chapter}/'
response = requests.get(url, headers = headers)
soup = BeautifulSoup(response.text, 'html.parser')
mangas = soup.find_all('img', class_="img-responsive")
chapter += 1
def get_manga_details(manga):
src = manga.find('img', class_= "img-responsive").find("img")["src"]
alt = manga.find('img', class_= "img-responsive").find("img")["alt"]
return {
"chapter": chapter,
"src": src,"alt": alt
}
all_mangas = [get_manga_details(manga) for manga in mangas]
with open("manga.json", "w") as write_file:
json.dump(all_mangas, write_file)
print("Success")
This code functions in cmd but produces empty output.
Which is wrong, please teach me
I want it to be:
{
"chapter": "number": 1[
{
"src": "here", "alt" : "here",
"src": "here", "alt" : "here"
}]
}
Please guide me

Answer 1

There's a lot of things that are wrong with your code. First, the URL you are trying to access returns a 404, you need to rjust the chapter number with leading zeroes. Second, your logic and loops don't make much of a sense like defining your function and lists inside the loop, then expecting the output to contain all the chapters. Moreover, you're calling BeautifulSoup's find function again in your function which is not needed, you can directly access the attributes.
See my code below, it works on my machine
import requests
import json
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Mozilla/5.0 (Linux; Android 4.3; nl-nl; SAMSUNG GT-I9505 Build/JSS15J) AppleWebKit/537.36 (KHTML, like Gecko) Version/1.5 Chrome/28.0.1500.94 Mobile Safari/537.36'}
chapter = 1
allmangas=[]
def get_manga_details(manga,i):
print(manga)
src = manga["src"]
alt = manga["alt"]
return {
"number": i,
"src": src,"alt": alt
}
while chapter < 3 :
url = 'http://www.komikgue.com/manga/one-piece/'+str(chapter).rjust(3,'0')
response = requests.get(url, headers = headers)
soup = BeautifulSoup(response.text, 'html.parser')
mangas = soup.find_all('img', class_="img-responsive")
print(mangas)
allmangas.append({'chapter':chapter, 'data':[get_manga_details(manga,i) for i,manga in enumerate(mangas[:-1])]})
chapter += 1
with open("manga.json", "w") as write_file:
json.dump(allmangas, write_file)
print("Success")