I'm trying to replicate a request I make on a website (ie zoominfo.com) using the same http POST parameters using chrome rest console, but it fails for some reason. I'm not sure if there is a missing field or it's not working because the origin of the request isn't valid.. can someone point me out in the right direction? Below is a detailed explanation of the experiment:
ORIGINAL CASE
basically if I go to zoominfo.com (registered and all) I see a form page that I need to fill:
if I hit enter.. the site makes an ajax call. If I open the chrome web dev tools, and open the network tab, I see the details of the ajax call:
notice the body of the POST has the name John Becker in it:
{"boardMember":{"value":"Include","isUsed":true},"workHistory":{"value":"CurrentAndPast","isUsed":true},"includePartialProfiles":{"value":true,"isUsed":true},"personName":{"value":"john%20becker","isUsed":true},"lastUpdated":{"value":0,"isUsed":true}}
the response is shown under the respones tag:
WHAT I'M TRYING TO DO
basically replicate what i've done above using a REST console (note: so there is nothing illegal here.. i'm just replacing a chrome browser action with a rest client action.. i'm not hacking anyone and i'm not getting information I can't get the normal way, but if someone feels otherwise.. please let me know)..
so I plug in the same parameters as above into the rest console:
now i'm not sure about authentication.. but just to be safe, i entered the same user name and pwd i have for the site into the REST console:
but then I keep on getting an error as a response to my rest console's request:
UPDATE: CORRECT ANSWER:
so according to JMTyler's answer.. I had to simply include criteria in the RAW body, and convert it to url encoding.. in addition to that, I had to explicitly set the encoding in the rest console body..
looking at the chrome inspector more closely, it turns out that I simply had to click on view source:
to get the url-encoded value that I needed to put in the RAW body in the rest console:
I also had to set encoding to gzip,deflate,sdch and things worked fine!
The form is posting all that JSON under the field criteria. You can see this in the screencap of the chrome dev console you posted.
Just start your raw body in rest console with criteria= and make sure the json has been url-encoded. That should do it.
No authentication is needed because none is passed through the headers in your screencap. Any cookies you have when you load the page normally will also be loaded through rest console, so you don't need to worry about explicitly setting them.
Reading your problems I'll make an educated guess:
zoominfo does not provide an RESTful API.
Rest-Console understands and uses HTTP Authentication, which is different from the authentication handler zoominfo implemented.
A possible way to work around may be:
Make a call to the login-page via rest console. you'll get back cookies and a lot more.
In subsequent requests to zoominfo be sure to include those cookies (likely holding some session information) in your request, therefore acting like a browser.
Related
I'm trying to get data from this site: [1] https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
I found this link where I can get the data in JSON format: [2] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
But there is a problem:
The JSON link Doesn't work every time in fact sometimes I get a 404 error.
I noticed that if I open the first link [1] before opening the second [2] it works perfectly.
This error is also more frequent when I try to scrape other data on the same site: [3] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio/piu-giocate/u-o-goal?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
In this link [3] I try to get all "u-o-goal" odds but this link works only if (before starting my program to scrape data) in the main link [1] I press the "U/O GOAL" button -> https://i.stack.imgur.com/Nei5u.png
In my code, I'm using Java and htmlunit to scrape the data.
My question is: how this webpage works, why couldn't I open directly the links [2]/[3], I know that there is a sort of request and approval system behind but I can't see where.
You cannot directly open these URLs since the website (and many like it) will use cookies and bot-prevention techniques/session tracking so they can gather data about usage of their website. eg. they set a "Referer".
I'm not going to code a solution for you but I can at least help you understand what you need to do to get to where you want...
I've attempted to summarise how I'd typically unpick a request like this to recreate it, but in its essence, you need to understand the sequence of HTTP requests being made (this is how the web works - HTTP requests).
First you typically start with no session cookies and you access the site directly (no referer).
Once you access a website, typically the server responds with a session cookie for you to communicate back to the server a unique session ID so it has some sort of record of your browser having already been in contact.
Your browser may make more requests (asynchronously) and in doing so typically sends the cookies and the referring URL (usually the base Url will work... just don't use something that starts with something other than "https://www.eurobet.it"
anything else you're going to need to figure it out. Lots of headers are optional. Lots of query params have defaults.
https://stackoverflow.com/a/64671815/7619034 - here's an answer I've given before that answers this type of question which comes up often enough.
so to explain a bit further, for your specific scenario...
When you access https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI, the server responds with HTTP headers:
...
set-cookie: __cfduid=dd38d***********41125; ...
...
The rest doesn't look that relevant:
Going straight to the other request: https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
This HTTP request takes (as input):
cookie: __cfduid=dd38d***********41125; mbox=session#6661556c.....b6e8cc1fa6f03#1608242987; at_check=true; s_ecid=MCMID%***********2021453010; AMCVS_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1; AMCV_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1075005958%7CMCIDTS%7C18614%7CMCMID%7C91883906030825914429183258312021453010%7CMCAID%7CNONE%7CMCOPTOUT-1608248327s%7CNONE%7CvVersion%7C4.4.1; s_cc=true
...
referer: https://www.eurobet.it/it/scommesse/
...
x-eb-accept-language: it_IT
x-eb-marketid: 5
x-eb-platformid: 1
Cookies are set in an initial request (typically) using Set-Cookie header and then are passed back to the server in subsequent requests using the cookie header.
I'm not certain how many of these values are relevant but you'd need to figure out where each came from in the chain of HTTP requests between the initial one and this one and you'd need to replicate them (see url above of my previous answer - warning this can be time consuming).
The other headers can be set statically most likely since they probably aren't due to change.
If you have access to curl on the command line, you can attempt to reconstruct some of these requests by hand. Some will be time sensitive since cookies do expire after an amount of time (see set-cookie header details for exactly when). Once you've reconstructed a working request, you can then start coding it in your application.
If you can work all this out you should be able to re-construct the chain of HTTP GET requests to get the JSON data you want. Good luck!
Can anyone tell me if it is possible to specify mime-type returned from a RESTful web service through the URL? I am trying to demo a simple service that I created to someone and I am just using a web browser (Chrome) to invoke the service. I am trying things like this:
http://localhost:8088/providers?mimeType={application/json}
http://localhost:8088/providers?mimeType=application/json
http://localhost:8088/providers?mimeType=json
None of these work for me - I just keep getting XML returned no matter what I select.
I found several posts related to this subject, but nothing with an answer about how to do this through a URL (I found some which talked about setting headers, but I am using a browser and don't have the ability to specify headers). I found this post:
REST Content-Type: Should it be based on extension or Accept header?
And that post linked to this: http://www.xml.com/pub/a/2004/08/11/rest.html
On the xml.com site, I found this text:
URI-Specified Representation [PS, AR]
A client can specify the representation using the following query string:
mimeType={mime-type}
A REST server should support this query.
So it seems that what I am trying to do should be possible, but I can't figure out how to make it work. Can anyone help?
Thanks.
To define the headers added to your request, you can use a Chrome extension like Postman.
Then you will be able to specify an Accept header to tell Talend to return json:
Accept: Application/json
I am trying to understand how POST and/or GET methods work in terms of the actual browser.
I am attempting to contact an API which requires API key, method you wish to use on their side, and an IP address at the minimum.
My original idea was to do something like this:
I feel like I'm on the right track, it does something and gets an error as opposed to telling me the page does not exist. I'm expecting either JSON or XML in response as the API supports both but instead I get this error:
This page contains the following errors:
error on line 1 at column 1: Document is empty
Below is a rendering of the page up to the first error.
Upon studying the documentation of the API more, I found something saying that methods are called using HTML form application/x-www-form-urlencoded and the resuource models are given as form elements.
I tried researching what that means to see what the problem was and found this site http://www.w3.org/TR/xforms11/ but I'm still unclear.
Ideas?
It seems to mean that the application is expecting a POST method but you're doing a request with a GET method (when you use the querystrings).
Since you can't just do browser requests using POST using the address bar, you may need to:
Construct a simple JS function that does a xmlhttprequest request using that method instead, and running it from the console;
Create a simple HTML page that automates the above process, allowing you do make POST calls;
Using CURL instead, which is a great tool for testing those kinds of requests.
I have a simple RESTful web service and I wish to test the PUT method on a certain resource. I would like to do it in the most simple way using as few additional tools as possible.
For instance, testing the GET method of a resource is the peak of simplicity - just going to the resource URL in the browser. I understand that it is impossible to reach the same level of simplicity when testing a PUT method.
The following two assumptions should ease the task:
The request body is a json string prepared beforehand. Meaning, whatever is the solution to my problem it does not have to compose a json string from the user input - the user input is the final json string.
The REST engine I use (OpenRasta) understands certain URL decorators, which tell it what is the desired HTTP method. Hence I can issue a POST request, which would be treated as a PUT request inside the REST engine. This means, regular html form can be used to test the PUT action.
However, I wish the user to be able to enter the URL of the resource to be PUT to, which makes the task more complicated, but eases the testing.
Thanks to all the good samaritans out there in advance.
P.S.
I have neither PHP nor PERL installed, but I do have python. However, staying within the realm of javascript seems to be the simplest approach, if possible. My OS is Windows, if that matters.
I'd suggest using the Poster add-on for Firefox. You can find it over here.
As well as providing a means to inspect HTTP requests coming from desktop and web applications, Fiddler allows you to create arbitrary HTTP requests (as well as resend ones that were previously sent by an application).
It is browser-agnostic.
I use the RESTClient firefox plugin (you can not use an URL for the message body but at least you can save your request) but also would recommend curl on the command line.
Maybe you should also have a look at this SO question.
New here. I've searched quite a bit for a working solution to my problem, but even though I have found posts with promising titles, none of the solutions have worked.
I am deploying an MVC2 web app to a client's server.
I did my development on Win2k8 Server, but they are running Win2k3 sever.
The app's only purpose is to receive some record ID information as HTTP parameters, check in the database for the status of the given record or records, and then return the status information as a simple string such as "Completed" or "Incomplete" in JSON format.
This getJSON code works fine in the development environment.
Inexplicably to me, on the client's server, the getJSON request receives a null response from the application.
There is no cross-domain action AFAIK... the result is the same from the client's server or from my machine via VPN.
In the MVC model's Json code, a common solution for people is to add the "JsonRequestBehavior.AllowGet" attribute to the Json result being returned. I did this long before trying to deploy it, and as I said, it has worked fine in the dev environment.
Using Firebug, I have watched the same request URL get sent to both my local server and the client server - the response headers from both servers are the same, but the response content from my server is shown as:
{"Result":"No Data"}
Which is what I want.
There is literally no content shown in the response from the client's server..? But the request gets an HTTP 200 code and is recorded as a success in the reponse's status attribute.
The response header content type in both situations is "application/json"
But wait, there is more!
If I manually enter the request to each server in the Firefox nav bar, and hit enter, in both cases it responds with:
{"Result":"No Data"}
Which is what I want. So why can I get the result I want from the MVC app on the client's server only when I hand-enter the request URL in Firefox, but not from the Javascript code?
I have tried forcing different output content types ... using the jQuery ajaxSetup method...
$.ajaxSetup({
async: false,
dataType: 'text'
});
or
$.ajaxSetup({
async: false,
dataType: 'html'
});
and again wtih 'script', and 'json'. I also tried the conversion options such as 'text json' or 'html json' or 'json text' and so forth.
Some of the posts I'm reading, and my gut feeling, though, suggest the problem is not the jQuery code making the request that is at fault... I don't see how the same jQuery request point to a different server running the same app would suddenly cause that server to send back a 'null' value.
By null, I want to be clear... I mean nothing is sent. There is no {} or {null} or any sign of JSON... just blank whiteness of non-existence :P
Even if nobody knows the answer, I would love some input perhaps suggesting where I should focus my sleuthing ... client or server?
If the problem is the server, it seems hard to really know that the MVC stuff is running 100% on the IIS6 server, but in general it seems to work. I have a different MVC app running on the client server which responds to the virtual paths, and generally runs the same as on dev machine.
I have found one thing ... the request headers are somewhat different? The Request Headers sent to the IIS7 setup include an "X-Requested-With: XMLHttpRequest", "referrer" , and "cookie" field/value.
I could guess that the lack of the "X-requested-with: XMLHttpRequest" in the IIS6 request headers is a clue, but I do not see then how the same javascript code pointing at a different server can generate different request headers itself. So how else are those being generated?
The javascript is embedded in an ASP.NET page, btw.
Oooh.. frustration!
Thanks for any input.
Odd Progress ... apparently there is some sort of issue with IIS6 handling the query. Although I have not payed any attention to JSONP, a post elsewhere suggested that sometimes use the "&callback=?" parameter at the end of a .getJSON request URL would force it into GET mode and that worked frequently for problems getting data from the server. So I did that... and it did work, sort of. The proper {"Result":"No Data"} was returned in response to the request... which seems good. However, the way that the JSONP callback works, it generates its own script to do the calling and fetching and interpreting of the incoming JSON. In this case, it interprets the JSON to need a label which it does not have, thus an error is thrown "invalid label" ... there must be some way to hack things to just deliver the JSON, but the whole required use of JSONP callbacks suggests that the server configuration is wrong, right? Or why does it work without JSONP for IIS7 and not IIS6?
Despite my not liking the callback JSONP solution, it appears to work ok. The error is still returned about an invalid label, but that does not appear to stop the remaining javascript from running... and so the application is working with IIS6 now. I have not tested the fix of using the callbacks and JSONP against IIS7 yet, but I expect it will work well enough.
Here is a link to the discussion that lead me to my current solution. I do still hope to find a more elegant solution, however.
NeoWin.net
Are you certain that your App Extension Mappings are set up correct?
Follow this article for running MVC2 on IIS6 and ensure all the different configurations have been done, that's probably the first step before going further and investigating specifics.
I'm really inclined to believe it's related to HTTP Verbs.