Is it possible to get the information presented in e.g. http://localhost:9222/json directly from Puppeteer without having to make the http request?
I do not remember what information exactly in http://localhost:9222/json but you can use browser.version() for learn browser version and you can learn browser.wsEndpoint() for browser endpoint url, without http request.
Anyone know the answer to this? I have a similar question, my problem is that I want to know where the page.goto(url) resolves
I have a code like this:
url = "http://www.somewebsite.com/"
response = await page.goto(url,
waitUntil=['domcontentloaded', 'networkidle0'], timeout=30000
)
Then I want to check when the url redirect, so I did it something like this
url_resolve = response.url
But the problem is I'm still getting the original url I requested, but checking http://localhost:9222/json, I'm getting something like this
[{
"description": "",
"devtoolsFrontendUrl": "/devtools/inspector.html?ws=localhost:9222/devtools/page/BA2493F44C89E9255BD0FE6B060AB0FE",
"id": "BA2493F44C89E9255BD0FE6B060AB0FE",
"title: "Some Website - Design",
"type": "page",
"url": "http://www.somewebsite.com/home/13112321/main.html"
"webSocketDebuggerUrl": "ws://localhost:9222/devtools/page/BA2493F44C89E9255BD0FE6B060AB0FE"
}]
So what I really want is to get the url I'm getting from the endpoint http://localhost:9222/json, which only accessable via http request not from puppeteer itself, btw I'm using pyppeteer
Update:
So I found a way to go around solving my use case, What I did was to use the evaluate method, by simply adding
current_url = await.page.evaluate('window.location.href', force_expr=True)
Then I'll get the current url of the open tab
Got the answer here
Related
I am trying to intercept a website - https://www.kroger.com/pl/chicken/05002. In the chrome network tab, I see the request as below, with the details of the products nicely listed as JSON
I copied the cURL as bash and imported it as raw text in Postman. It ran forever without any response. Then I used the intercept feature and still it is running forever.
When both the requests are exactly same, why is it running in Chrome and not in Postman? What am i missing? Any help is appreciated, thanks in advance.
This is probably happening because they don't want you to do what you are trying to do. Note the "filter.verified" param in the URL.
You may want to try reaching out to them for an external API token - especially if you are creating an app or extension to compare competitive prices with the intention of distributing said app or extension - regardless of if it is for financial compensation or not.
Ethically questionable workaround (which would defintely need to be improved upon - this is simply an example of how you could solve your problem...):
GET https://www.kroger.com/search?query=chicken&searchType=default_search&fulfillment=all
const html = cheerio(responseBody);
var results = [];
html.find('div[class="AutoGrid-cell min-w-0"] > div').each(function (i, e)
{
results.push({
"Item": e.children[e.children.length-3].children[0].children[0].children[0]["data"],
"Price": e.children[e.children.length-4].children[0].attribs["value"]
})
});
console.log(results);
If you are unable to obtain an API token from them, this would probably be a legal way to accomplish what you want.
I started using API.AI and Dialogflow in its first versions for some small time projects.
Recently I wanted to try and dive into the new V2 of Dialogflow and see how I can continue to build nice Google Assistant apps with that.
When trying to formulate a response (based on the documentation here https://dialogflow.com/docs/reference/api-v2/rest/v2beta1/WebhookResponse) I am unable to actually render responses of any kind. Everytime I do it just gives me a webhook error back.
The intent That I'm using in my demo project is (still fairly simple as I'm just trying to get a response back):
My Webhook (Elixir based) returns the following response (actual production response):
When inspecting the "Show JSON" After doing the test on the right-hand side of the Dialogflow screen I receive:
I must be doing something wrong, should the whole response that I send now be wrapped in something?
Update:
When removing "fullfillmentText" and just keeping "fullfillmentMessages" I seem to get the same error, but then for fullfillmentMessages. It looks like DialogFlow doesn't understand the JSON parameters I'm sending to it. example:
Man, what a typo here... Managed to fix it in the end by writing "fulfillmentMessage".
Protip for everyone starting with this and wanting to know the structure of data:
Make a simple intention, just as a test
Add some google or other responses trough the GUI
Save the intention
Trigger the intention from the "tryout now" function on the right-hand side.
Click SHOW JSON to inspect how a response would need to look.
Final result Code sample:
{
"fulfillmentMessages": [
{
"platform": "ACTIONS_ON_GOOGLE",
"simpleResponses": {
"simpleResponses": [
{
"displayText": "Sorry, something went wrong",
"ssml": "<speak>Sorry, Something went wrong <break time=\"200ms\"/> Please try again later</speak>"
}
]
}
}
]
}
I've been researching this and cannot find or understand some of the solutions so i'm hoping to get some help here. I'm using Asp.net and building an application that needs to use a bible api. I like the two listed in the question. Every time I call esvapi it comes back successful, but I cannot view the data. I get an error in the console.
XMLHttpRequest cannot load http://www.esvapi.org/v2/rest`/passageQuery?key=8834092f0c58fcda&passage=James2. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:59324' is therefore not allowed access.`
I've seen other with this error and I have questions.
If I'm understanding this correct I get this because the server is preventing me from seeing the data for security purposes. Maybe even the browser( this is not just a chrome issue) problem. So if I need to add a info to the response header from Angularjs to stop this how is that done. Anyone with experience?
Would I need to contact anyone to be able to prevent the server from responding this way...I doubt this, but thought I would ask. I already have valid api key.
the bible.org website api key is confusing to apply to my code. on esvapi i just add a header with key: "keypass" and I only have the CORS issue. But with bible.org I can't figure out how to implement the api key and password. see below... Do I say token:key: username. If i put the api in the browser I get a popup to add username and password. the username is my key and the password is ignored. I tried putting in username as key, but that didn't cut it. Regardless I need to fix the CORS issue and add info to response headers to see response data.
$scope.search = function() {
return $http.get("http://www.esvapi.org/v2/rest/passageQuery?&passage=" + $scope.bo + $scope.chap, {
headers: {
"key?token?orusername?": "",
///thought i saw someone do this...don't know if this is right
"Access-Control-Expose-Headers": "Content-Disposition",
}
}).success(function (data, status, headers, config) {
$scope.book = data.Book;
$scope.chapter = data.Chapter;
$scope.output = data.Output;
}).error(function (data, status, headers, config) {
$scope.message = "Oops... something went wrong";
});
Any input would be helpful. Thanks!
I actually have a bible api working...just a version that I don't like and there is not another version on that webites api.
Change the get $http.get call to $http.jsonp and hope it works. You're using cross-site scripting. Sometimes you can get away with a JSONP call in these cases and sometimes you can't.
I am a novice to Angularjs and tried to follow example given for $http.get on angularjs website documentation.
I have a REST service, which when invoked returns data as follows:
http://abc.com:8080/Files/REST/v1/list?&filter=FILE
{
"files": [
{
"filename": "a.json",
"type": "json",
"uploaded_ts": "20130321"
},
{
"filename": "b.xml",
"type": "xml",
"uploaded_ts": "20130321"
}
],
"num_files": 2}
Part of the contents of my index.html file looks like as follows:
<div class="span6" ng-controller="FetchCtrl">
<form class="form-horizontal">
<button class="btn btn-success btn-large" ng-click="fetch()">Search</button>
</form>
<h2>File Names</h2>
<pre>http status code: {{status}}</pre>
<div ng-repeat="file in data.files">
<pre>Filename: {{file.filename}}</pre>
</div>
And my js file looks as follows:
function FetchCtrl($scope, $http, $templateCache) {
$scope.method = 'GET'; $scope.url = 'http://abc.com:8080/Files/REST/v1/list?&filter=FILE';
$scope.fetch = function() {
$scope.code = null;
$scope.response = null;
$http({method: $scope.method, url: $scope.url, cache: $templateCache}).
success(function(data, status) {
$scope.status = status;
$scope.data = data;
}).
error(function(data, status) {
$scope.data = data || "Request failed";
$scope.status = status;
});
};
}
But when I run this, I do not see any result for filenames and I see http status code = 0
When I run , http://abc.com:8080/Files/REST/v1/list?&filter=FILE in browser, I still can see desired results (as mentioned above)
I even tried to debug using Firebug in firefox, I see the above URL gets invoked when I hit "Search" button but response looks to be empty. And interestingly in Firebug under URL, it shows
OPTIONS "Above URL"
instead of
GET "Above URL"
Can you please let me know, what I am doing wrong and why I am not able to access JSON data ?
Thanks,
This is because how angular treats CORS requests (Cross-site HTTP requests). Angular adds some extra HTTP headers by default which is why your are seeing OPTIONS request instead of GET. Try removing X-Requested-With HTTP header by adding this line of code:
delete $http.defaults.headers.common['X-Requested-With'];
Regarding CORS, following is mentioned on Mozilla Developer Network:
The Cross-Origin Resource Sharing standard works by adding new HTTP
headers that allow servers to describe the set of origins that are
permitted to read that information using a web browser.
I have been having the issue using $resource, which also uses $http.
I noticed that when I used AngularJS 1.0.6 the request would not even show up in Firebug, but when using AngularJS 1.1.4 Firebug would show the GET request and the 200 OK response as well as the correct headers, but an empty response. In fact, the headers also showed that the data was coming back as shown by the "Content-Length" header having the correct content length, and comparing this against a REST Client plugin I was using that was successfully retrieving the data.
After being even further suspicious I decided to try a different browser. I had originally been using Firefox 16.0.1 (and also tried 20.0.1), but when I tried IE 9 (and AngularJS 1.1.4) the code worked properly with no issues at all.
Hopefully this will help you find a workaround. In my case, I noticed that I never had this problem with relative URLs, so I'm changing my app around so that both the app and the API are being served on the same port. This could potentially be an AngularJS bug.
I had the same problem today with firefox. IE worked fine. I didn't think it was cors at first because like you I got no errors in the console and got a status of 0 back in my error method in angular. In the firefox console I was getting a 200 response back in my headers and a content length, but no actual response message. Firefox used to give you a warning about cross site scripting that would point you in the right direction.
I resolved the issue by setting up cors on my api. This is really the best way to go.
If you are only using GET with your api you could also try using jsonp this is built right into angular and it is a work around for cors when you do not control the api you are consuming.
$http.jsonp('http://yourapi.com/someurl')
.success(function (data, status, headers, config) {
alert("Hooray!");
})
.error(function (data, status, headers, config) {
alert("Dang It!");
});
It's cross-site-scripting protection.
Try starting google chrome with --disbable-web-security (via command line).
If that isn't working also try to put your angular stuff into an http server instead of using the file protocol. (Tip: use chrome canary if you want to have a browser dedicated to --disable-web-security - of course you'll have to set the command line argument too, but both chrome versions run simultaneously). For release you'll have to set some http headers on the server providing the AngularJS-stuff to allow access to the twitter api or whatever you want to call.
What I'm trying to do is to get JSON data (whatever it is just make sure I can get anything is ok) from Rallydev without login to make sure that Rallydev is on. I tried several ways, but each way requires a username and password. Would anyone provide a URI for this? Thank you, Guys. Anything you give would be appreciated.
String url = "link";
Client client = Client.create();
WebResource webResource = client.resource(url);
String s = webResource.get(String.class);
try {
// check if RallyDev service is up
if (s.contains("Hello it is now "))
_log.information("RallyDev is working... at " + new Date());
} catch (Exception e) {
_log.error(ErrorCodeEnum.INTERNAL_ERROR, "RallyDev service might be down!!! in " + new Date(), e);
}
Ok, so whether it's Jersey client or plain old HTTP GET against Rally, I'm coming back to my original comment that developing a Java app to do this is a bit overkill. You could accomplish the same thing with a one-line curl command in Linux:
curl -u 'rallyuser#company.com:rallypassword' https://rally1.rallydev.com/slm/webservice/1.33/hierarchicalrequirement.js?pagesize=1
A valid (Rally is up and responding) response to this might look like:
{"QueryResult": {"_rallyAPIMajor": "1", "_rallyAPIMinor": "33", "Errors": [], "Warnings": [], "TotalResultCount": 84, "StartIndex": 1, "PageSize": 1, "Results": [{"_rallyAPIMajor": "1", "_rallyAPIMinor": "33", "_ref": "https://rally1.rallydev.com/slm/webservice/1.33/hierarchicalrequirement/12345678910.js", "_refObjectName": "My Story Name", "_type": "HierarchicalRequirement"}]}}
Looks like you're using Jersey Client to setup a REST connection in Java. Are you really needing to do this without providing credentials? You'll need to pass credentials of some sort as any query-able endpoint in Rally is going to require HTTP Basic Authentication.
If you are looking for the appropriate REST syntax and endpoints to formulate a valid query, you may wish to look at our Webservices API documentation on REST queries:
https://rally1.rallydev.com/slm/doc/webservice/rest.jsp
As an example, a valid REST URL to do a query and get back JSON-formatted results is as follows. A GET against the following sample URL queries user stories owned by user: user#company.com:
https://rally1.rallydev.com/slm/webservice/1.33/hierarchicalrequirement.js?query=(Owner = "user#company.com")&pagesize=1
This would return the first matching User Story.
A simple example of accessing Rally REST services in Java is here:
http://www.rallydev.com/help/basic-rest-client-operations-java
And a full (alpha-release) Java REST API toolkit for Rally is here:
http://www.rallydev.com/developer/java-toolkit-rally-rest-api
All of the above seems like a bit overkill for just finding out if the Rally service is up. You can subscribe to Rally's status updates via RSS at http://status.rallydev.com as a good way to stay apprised of system status information.
I hope this helps - if the answer is off-target, please provide some further clarifying comments and we'll do our best to answer.