youtube-dl: Failed to parse JSON - json

Question

I posted a question on this yesterday to the GitHub support page and it got flagged as a duplicate - the original answer is here. This issue doesn't automatically fix like it did for that user. Instead, it seems to come and go with no pattern, so I don't have a good way to replicate it. Some songs will work at one point in time, then they won't a couple minutes later.
Error:
[debug] Encodings: locale cp1252, fs utf-8, out UTF-8, pref cp1252
[debug] youtube-dl version 2020.09.20
[debug] Python version 3.7.8 (CPython) - Windows-10-10.0.19041-SP0
[youtube:search] query "iron man 3 song": Downloading page 1
[debug] exe versions: none
[debug] Proxy map: {}
ERROR: query "song name": Failed to parse JSON caused by JSONDecodeError('Expecting value: line 1 column 1 (char 0)')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
I get the issue when attempting to extract the data from the video. Here is a snippet of the code I am using:
ydlOps = {
'format': 'bestaudio/best',
'outtmpl': './%(title)s.webm',
'noplaylist': True,
'extractaudio': True,
'audioformat': 'webm',
'default_search': 'ytsearch1',
'quite': True,
'verbose':True,
'version': True
}
with youtube_dl.YoutubeDL(ydlOps) as downloader:
songData = downloader.extract_info(url, download=download)
I have changed the options, tried other options said to have worked, and nothing seems to make a difference. Some will work, then not, then they will again.

Answer 1

I think this is a youtube-dl bug. I wrote a parser for youtube searches and it also broke.
When looking previously at the response from youtube, all the JSON data was stored like this:
window["ytInitialData"] = {...}
So you just had to search through the server's response for the string 'window["ytInitialData"]' to find the relevant JSON and extract it. But now, youtube stores the JSON like this inside the html file sent by the server
var ytInitialData = {...}
This needs to be changed on youtube-dl's part when parsing the results.
What's strange is that sometimes youtube uses the previous version and sometimes it uses the current one. I think it's because the change in javascript is progressively being rolled out accross all youtube servers.
Also note that now, the line containing all the JSON ends with '; ' instead of just ';'. This might also require a change from youtube-dl.
You need to submit a pull request to youtube-dl or wait for somebody to fix it.