Are there any ways to Copy as cURL (bash) with puppeteer or playwright?
Not directly. As suggested by the commenter, you can easily write something to take a route to curl. A route encapsulates both the request and subsequent response within playwright.
However, there is also an easier, more direct way by leveraging the HAR file format.
Tell playwright to record a .har, optionally filtering to only a subset of URLs you care about.
Load the .har into a HAR -> cURL converter tool like this.
Related
In Chrome Dev tools/Network tab I have filtered the requests using filter like "-.json -.png -.woff2".
I want to export the remaining requests (those which are visible on the screen).
However when I use "Export HAR..." icon, in the output HAR file I still get all the requests including the hidden.
Is there a way to do this?
Thanks in advance
This is currently not possible in Chrome DevTools:
DevTools saves all requests that have occurred since you opened DevTools to the HAR file. There is no way to filter requests, or to save just a single request.
Ref. https://developer.chrome.com/docs/devtools/network/reference/#save-as-har
However, since a HAR file is just a JSON, you can use something like jq to extract the requests you are interested in.
Let's say that you exported a HAR file from a Wikipedia page and called it wiki.har. Here is how you can create a HAR file containing only the requests for the images on that page:
cat wiki.har | jq 'del(.log.entries)' > wiki-with-no-entries.json
cat wiki.har | jq '.log.entries | map(select(._resourceType == "image"))' > image-entries.json
cat wiki-with-no-entries.json | jq '.log += {"entries": $myEntries}' --argfile myEntries image-entries.json > wiki-with-image-entries.json
Note 1: _resourceType can be document, image, font, stylesheet, script, other (maybe more? I could not find any documentation about this).
Note 2: the jq documentation recommends using --slurpfile instead of --argfile. However, --slurpfile here would create entries:[[]], while --argfile creates entries:[] (which is what you need). This is probably just me not knowing too much jq though...
See also:
Create a har file with xhr request only in chrome dev tools
You could give webQsee (https://webqsee.com) a try. It allows you to export filtered events. HAR format is one option. Additional options are Excel (also contains console outputs) and "Behavior Reports". They combine network and console output with the screen video.
Is there a way to create a har file with xhr request only in chrome dev tools?
Even though I can filter the requests in dev tools, when I save it as a har file, it contains all the requests.
If the dev devtools doesn't support that, is there a alternative way to do this?
Seems like there is no direct way to filter the requests. What I did was, creating the har file with all the requests and filter that file, then save it again after removing unwanted requests.
There is a entry called entries in the content inside har file. It contains all the requests we have sent in an array. I got the har file content and filtered it using JSONPath expressions.
About JSONPath, JSONPAth evaluator
Expression to filter application/json types : $..entries[?(#.response.content.mimeType == "application/json")]
After that I have replaced the entries array with the one I have filtered in the har file and saved it again. This way you can get a har file with filtered requests. Same way you can filter XHR or any other types using the relevant JSONPath expression
I recently came across this same issue while doing some research for mocking network calls in playwright. It does provide ability to record only the call you are interested in. Here is the link:
https://playwright.dev/docs/network#recording-har-with-cli
I'm working on functional testing of a REST API using Jmeter and JSON. I've got file uploading working but I cant seem to get the file downloading to work in jmeter. I'm saving the response as noted in this post: JMeter - File upload and file download scenario
When I do this, I am getting close but not quite exactly what I need. This is and example of what I am getting:
--0rVAdzesdQq7VrwJaRoYGm_UHdMD5nhi9_5w4u
Content-Disposition: form-data; name="api-test"; filename="LIBFILE1.txt"
Content-Type: text/plain
Content-Transfer-Encoding: binary
Library File for Automated Smoke Test
This file is used to test file library upload for automated smoke tests.
--0rVAdzesdQq7VrwJaRoYGm_UHdMD5nhi9_5w4u--
How can I get the download to only save the file contents? In this example, it should only be:
Library File for Automated Smoke Test
This file is used to test file library upload for automated smoke tests.
This is a simple textfile. I would also like to download the other formats that I'm uploading including jpg, png, docx, pdf but it wont be right if it has this extra data coming with it. I figure once I get a simple text file working, it will help me get the more difficult file types.
Well, Save Responses to a file listener will store the data you can observe in the "Response Data" tab of the View Results Tree listener. If you see these Content-Type, Content-Disposition, etc. headers as response data - most probably your upload wasn't successful as you should not be getting the headers this way.
I would recommend double checking that the same request being executed via browser or i.e. Postman tool returns the same response and fix your JMeter script in case of differences.
See Performance Testing: Upload and Download Scenarios with Apache JMeter article for details on how to properly mimic file operations with JMeter.
Alternative way of saving response data into a file is using JSR223 Listener, given you select "groovy" in the language dropdown you should be able to save the response using the following simple script:
new File("/path/to/your/file.txt").setBytes(prev.getResponseData())
I'd like to recursively download JSON resources from a RESTful HTTP endpoint and store these in a local directory structure, following links to related resources in the form of JSON strings containing HTTP URLs. Wget would seem to be a likely tool for the job, though its recursive download is apparently limited to HTML hyperlinks and CSS url() references.
The resources in question are Swagger documentation files similar to this one, though in my cases all of the URLs are absolute. The Swagger schema is fairly complicated, but it would be sufficient to follow any string that looks like an absolute HTTP(S) URL. Even better would be to follow absolute or relative paths specified in 'path' properties.
Can anyone suggest a general purpose recursive crawler that would do what I want here, or a lightweight way of scripting wget or similar to achieve it?
I ended up writing a shell script to solve the problem:
API_ROOT_URL="http://petstore.swagger.wordnik.com/api/api-docs"
OUT_DIR=`pwd`
function download_json {
echo "Downloading $1 to $OUT_DIR$2.json"
curl -sS $1 | jq . > $OUT_DIR$2.json
}
download_json $API_ROOT_URL /api-index
jq -r .apis[].path $OUT_DIR/api-index.json | while read -r API_PATH; do
API_PATH=${API_PATH#$API_ROOT_URL}
download_json $API_ROOT_URL$API_PATH $API_PATH
done
This uses jq to extract the API paths from the index file, and also to pretty print the JSON as it is downloaded. As webron mentions this will probably only be of interest to people still using the 1.x Swagger schema, though I can see myself adapting this script for other problems in the future.
One problem I've found with this for Swagger is that the order of entries in our API docs is apparently not stable. Running the script several times in a row against our API docs (generated by swagger-springmvc) results in minor changes to property orders. This can be partly fixed by sorting the JSON objects' property keys with jq's --sort-keys option, but this doesn't cover all cases, e.g. a model schema's required property which is a plain array of string property names.
If I use wget, and get an html file from a URL. How can I generate a HAR file from that HTML file.
Any open source implementations for generating har files from html files?
Once the HAR file is generated I can read data from the HAR file using harlib.
If possible please suggest C, C++ or Java implementations.
The primary point of the HAR format is to have a standard HTTP tracing format that many tools can use and analyze. In other words, it's original intent was and primarily is, for performance analysis, not "archiving" webpages per se.
If you fetch a page with wget, you're missing 99% of all the performance data. To capture the necessary data you really need a browser to execute the requests, fetch all the associated resources, save all the timers, etc. This will enable you to build the waterfall charts, etc.
If you need to capture this data on the server, then you can use pcap to capture the TCP trace and then convert that to HAR, although you still need a client which will actually parse the HTML and request all the sub-resources (pcap is just listening in the background). Alternatively, you can route your browser through a proxy and let it spit out a HAR file for you.
Last but not least, you can just drive the browser through its debug interface and export the HAR file that way. Java example for driving Firefox: https://github.com/Filirom1/browsermob-page-perf