Sometimes my HTTP responses have rather large content (maybe about 5 MB) and I want to save those together with the rest of the requests in the HAR file. However, the HAR file sometimes only contains part of the response content and sometimes only headers etc but nothing of the content at all. I tried most recent Chrome, Chromium and Firefox browsers.
To reproduce: Open developer tools, record, open a site that returns a large HTTP response (like a big photo JPEG or something) and save the HAR file. It will be much smaller than the size of the JPEG image.
Where is the setting so I can make the developer tool save the full response with the HAR file?
*** Disclaimer: Answer by HttpWatch Developer ***
Our HttpWatch software (Windows only) that works with Chrome and Edge can help you do this. Its default file format (HWL) contains a complete log of all responses whatever their size. The log files can be opened for visual inspection in HttpWatch Studio or examined programmatically using the HttpWatch API.
It can export to HAR files without any restriction on response body size but currently this is only possible using the ExportHAREx automation method:
https://apihelp.httpwatch.com/HttpWatch_API~HttpWatch.Log~ExportHAREx.html
If you save HAR files directly from the HttpWatch browser extensions response bodies are limited to 50 kB.
You can try HttpWatch for free:
https://www.httpwatch.com/download/
or contact support#neumetrix.com for more information
Here is a partial answer for Firefox, at least.
This is the bug report, which induces the patch: https://bugzilla.mozilla.org/show_bug.cgi?id=1223726 Open about:config and change devtools.netmonitor.responseBodyLimit from default value 1048576 to your desired size in bytes.
Related
For example, I know I can use Chrome's "Export HAR..." feature, but if I had the page open for an hour or so, "Export HAR" only saves the contents of the more recent requests, where the archive files representing earlier requests are actually empty. Maybe it depends on the size of the responses, but my archive ended up being 130MB in size and that definitely wasn't all of it.
*** Disclaimer: HttpWatch is a commercial tool and this was posted by its creator Neumetrix Limited ***
With HttpWatch you can automatically record and save to HAR files at regular intervals from Chrome:
https://help.httpwatch.com/automatic_recording.html
You can also use the HttpWatch log file format (HWL) resulting in much smaller log files and then export to HAR files at a later time.
I'm not familiar with web development but I believe this web page text content
https://almath123.github.io/semstyle_examples/
is made of two JSON files mentioned in it (semstyle_results.json and semstyle_results.json) and the JSON files are completely present in ram (If this is the correct term for referring to it) because when I disconnect the internet I can still browse the page and see the text content.
I want to download semstyle_results.json file. Is that possible? how can I do that?
Technically if you visit a website you're "downloading" the content. Your browser sends a request for information and a server responds by sending you the information. You're viewing that information locally. Dynamic sites poll or make further requests as you browse to keep the data updated and relevant, but it's sent to you.
If you want to easily download any of the content from the website, a simple way is to open up the development tools (CTRL + SHFT + I on windows for Firefox and Chrome), go to a source file and click save as. The network tab shows you requests that were made which includes not just files such as json but also the details of the request.
Here is a screenshot locating one of the json files in a Chrome-based browser (Brave)
The webpages may not always show that they will support json or xml return of data. For example if you inspect this webpage SEC EDGAR database using the method described above, it shows no json link but if you append index.json at the end of the link it will return the same data in json format or xml format, if you so please.
i.e: same website but with json endpoint
So it is always a good idea to see if the website hosts developer information. For example SEC EDGAR provides developer tools that mentions that the directory structure can be accessed via HTML, XML or JSON.
SEC developer information
Current Network Devtools tab displays MBs:
Is there some settings to display MBs in KBs and in full number, not decimal (33.5 as 33500)? Need to know precise value (e.g. 33.5 => 33527)
I don't think there is a way to change the unit used by the Size column, but you can do one of the following:
download the file and inspect its size locally
add a column for a 'Content-Length' response header
As you can see on the screenshot it's not perfect though - it's provided by the server and it's not required.
export entry as a HAR file and look up the exact value
use the Resource Timing API to list all resources in the console
(see Possible to list external resources loaded on a webpage with JavaScript? )
I want to use the function 'Preserve log' at the network tab in the Chrome Dev Tools. I searched internet and found following site. However, it does not work as it indicates. Does anyone have used preserve log and generate HAR file from it? Actually, I am wondering about the exact function of 'preserve log' itself.. Is is for generating HAR file?
https://community.box.com/t5/Managing-Content-Troubleshooting/How-to-Generate-a-HAR-File-in-Chrome-IE-or-Firefox/ta-p/366
Could you please help me out with this?
"Preserve log: option this is not the same like saving HAR file. You need preserve log in order to see the log even you refresh the page or move to another page. However you can't save full HAR file if this check box unchecked. So "preserve log" is required for HAR but it has other meaning.
* Disclaimer: Posted by Developer of HttpWatch *
HttpWatch on Windows can record the network traffic generated when a new Chrome tab or window is opened by enabling auto-recording in Tools->Options->Recording. In the new window click on the HttpWatch icon to view the network trace.The log file can either be saved a HAR or in the native HttpWatch format (HWL)
The free version will record a full HWL file that can be opened by someone using HttpWatch Professional.
* Disclaimer: Posted by Developer of HttpWatch *
I'm trying to take my entire website offline via the HTML5 FileSystem api. So far, I have has no problems, except for the Roboto.woff2 font that I have gotten from Google's website. Basically I have a blob that writes to a file in persistent storage. However, whenever I try to load the actual page, the console says: Failed to decode downloaded font: filesystem:http://localhost/persistent/fonts/roboto.woff2
1 OTS parsing error: Size of decompressed WOFF 2.0 is set to 0.
When I view the actual file in filesystem:http://localhost/persistent/fonts/, the file size is different as well. I suspect that it has to do with text/plain type. I've tried using application/x-font-woff, application/x-font-woff2, font/woff2, etc, but nothing has worked thus far. Any help would be appreciated!
So here's what I ended out doing. There are 2 ways.
Use charset=x-user-defined when opening the XMLHttpRequest
Use xhr2 (XMLHttpRequest with advanced features) and save as a blob. Here's a helpful link: xhr2
Option 2 works on all browsers except Opera Mini.