I'm trying to connect to ExactOnline server using HttpWebRequest from C#. When I try to get the Response, I get an exception: "The remote server returned an error: (400) Bad Request".
The Web Request looks like:
Method: GET
Address: "https://start.exactonline.nl/api/v1/3175257/Logistics/Items?$select=Code&$top=1"
Accept: application/json
ContentType: application/json
Authorization: "Bearer access_token"
where access_token looks like:
"stampNL001.gAAAAGivCOkntSKiT0xYatuOkLEkbA0cCcPAbdDZGctQSAHRuaJ1KfvMY1QjnKWLM4BnRNRh8Vpg9H-3ISW6Vs1Xr0EXjHxgxH1o-n4BJAySMw1tCF-v9heoQ_vQjS2zz8SZtYj1OT9U8DSJnvKzdd6dVKN90G3NA6k80EiS95wgxsVSBAIAAIAAAADO4MGzvH-iyio7XsXArprV_ey-zH9H-NPT2n4CBbjlIJ8gIkjLFvXrcJrZ2lwUBFOrgaHQwfU8dvmnSyRRzlZEe9wSfcpX16BPB7tZzrR_mdQozAtgWVxtIdzxUIHlqaFk0BNhOIfMdDxnagivTdo3HNdTVg9N8K0lx-TX4aNeeoRgzMho46Z1ix1te6rJ8_GjJeAjl7iyVDYqoK_D2Zlaa6cIYNillNlaOYxV2e95tcKoMLPRKUx3ULBtht_joijvA8raWhNBxHiJZQsIyCbTCJuC-dARqicrbdOqNkv769oRgnhLokWHt44dLpwQJ990eWqj1R6ppmF-W5s6d5EpQsLqkFSiPtpIHkao3D4Yxv6BCD8bhsjfjwAiISyyIPt7GbVv4OPZ7dDTMBZbWJBX2JLPWsxiPqb1Y1dOUPMxfFty9mM22qBXq8VA3EyA96-JwNqgIy4eP5hbXmeEU-BOxnF4vp_dZEZU-iM5fV-uYjZYduVtMNBHW-ubQZ811_rv1trx0TP7eEz8dbcfNlB0uAcb6NR-5tC2qwV0wb59qOjO2HQhb0TKGslPjefjwyhNK4ZVSWL0Cr_1KzxpKjA1suY12gBv_J6vQ4js3dlW1MxwypJaUzMMBvtGPqS2N3zcLvrMth1wiB7IjxfA5jd3hRo5_F3iCLTeDtLxToKpNA"
The same code (same input) worked two weeks ago.
What do I do wrong? Thanks.
The answer is in the response of this endpoint, if you query it without $filter. look at the following screenshot https://imgur.com/X4ufb94 it shows your endpoint call. it gives an error 400 bad request and states the answer to your problem: $filter is required for this endpoint.
Now look at this screenshot with $filter added https://imgur.com/c7fiGTx it gives back data without errors. i don't have any logistics data to show but it shows i don't have any and no error.
More in depth:
It looks like $filter has started to become an enforced required addition to certain endpoints. It was stated as 'required' for over a year, but just recently they started to actively block queries without the $filter parameter (speaking from experience).
From the release notes from august 2021:
Mandatory filtering for properties on 14 REST API endpoints To help
keep API traffic in Exact Online running efficiently, we have made
filtering mandatory for several properties within 12 API endpoints.
Filtering helps ensure that only relevant data is retrieved when you
make API calls, so you don’t have to work with a large amount of data.
PS. I post this as a new answer since my original answer was swiftly hidden end subsequently deleted (within an hour) by many mods. Corrections/changes to that answer on the other hand are not reviewed as spediently. Still waiting a day later. Since i think i am correct in my answer (i recently had to deal with the same issue on some code that had been running just fine for many months) i post it as a separate and new answer, for anyone looking for a real answer to the same issue.
Related
I'm trying to get data from this site: [1] https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
I found this link where I can get the data in JSON format: [2] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
But there is a problem:
The JSON link Doesn't work every time in fact sometimes I get a 404 error.
I noticed that if I open the first link [1] before opening the second [2] it works perfectly.
This error is also more frequent when I try to scrape other data on the same site: [3] https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio/piu-giocate/u-o-goal?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
In this link [3] I try to get all "u-o-goal" odds but this link works only if (before starting my program to scrape data) in the main link [1] I press the "U/O GOAL" button -> https://i.stack.imgur.com/Nei5u.png
In my code, I'm using Java and htmlunit to scrape the data.
My question is: how this webpage works, why couldn't I open directly the links [2]/[3], I know that there is a sort of request and approval system behind but I can't see where.
You cannot directly open these URLs since the website (and many like it) will use cookies and bot-prevention techniques/session tracking so they can gather data about usage of their website. eg. they set a "Referer".
I'm not going to code a solution for you but I can at least help you understand what you need to do to get to where you want...
I've attempted to summarise how I'd typically unpick a request like this to recreate it, but in its essence, you need to understand the sequence of HTTP requests being made (this is how the web works - HTTP requests).
First you typically start with no session cookies and you access the site directly (no referer).
Once you access a website, typically the server responds with a session cookie for you to communicate back to the server a unique session ID so it has some sort of record of your browser having already been in contact.
Your browser may make more requests (asynchronously) and in doing so typically sends the cookies and the referring URL (usually the base Url will work... just don't use something that starts with something other than "https://www.eurobet.it"
anything else you're going to need to figure it out. Lots of headers are optional. Lots of query params have defaults.
https://stackoverflow.com/a/64671815/7619034 - here's an answer I've given before that answers this type of question which comes up often enough.
so to explain a bit further, for your specific scenario...
When you access https://www.eurobet.it/it/scommesse/#!/calcio/?temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI, the server responds with HTTP headers:
...
set-cookie: __cfduid=dd38d***********41125; ...
...
The rest doesn't look that relevant:
Going straight to the other request: https://www.eurobet.it/detail-service/sport-schedule/services/discipline/calcio?prematch=1&live=0&temporalFilter=TEMPORAL_FILTER_OGGI_DOMANI
This HTTP request takes (as input):
cookie: __cfduid=dd38d***********41125; mbox=session#6661556c.....b6e8cc1fa6f03#1608242987; at_check=true; s_ecid=MCMID%***********2021453010; AMCVS_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1; AMCV_45F10C3A53DAEC9F0A490D4D%40AdobeOrg=1075005958%7CMCIDTS%7C18614%7CMCMID%7C91883906030825914429183258312021453010%7CMCAID%7CNONE%7CMCOPTOUT-1608248327s%7CNONE%7CvVersion%7C4.4.1; s_cc=true
...
referer: https://www.eurobet.it/it/scommesse/
...
x-eb-accept-language: it_IT
x-eb-marketid: 5
x-eb-platformid: 1
Cookies are set in an initial request (typically) using Set-Cookie header and then are passed back to the server in subsequent requests using the cookie header.
I'm not certain how many of these values are relevant but you'd need to figure out where each came from in the chain of HTTP requests between the initial one and this one and you'd need to replicate them (see url above of my previous answer - warning this can be time consuming).
The other headers can be set statically most likely since they probably aren't due to change.
If you have access to curl on the command line, you can attempt to reconstruct some of these requests by hand. Some will be time sensitive since cookies do expire after an amount of time (see set-cookie header details for exactly when). Once you've reconstructed a working request, you can then start coding it in your application.
If you can work all this out you should be able to re-construct the chain of HTTP GET requests to get the JSON data you want. Good luck!
What are the good choice for API error code response pattern?
Instead of using different codes indicating different type of error
100001 // username not provided
100002 // password not provided
100003 // password too short
...
I see some other use patterns like the following (non-sequential) ...
20000
20001
20004
20015
Are there any other recommendations?
In my experience developing and using web services, I have found that a strategy of using a combination of top-level HTTP status codes and lower level API error codes work reasonably well. Note that the lower level API error codes don't need to be integers, but can be any enumeration. For a well-known public example, AWS Simple Email Service (SES) uses this strategy of using both HTTP status codes and API level error codes. You can see a sample error code response for SES here. Note that although SES uses XML response error payloads, this strategy works equally well for JSON response payloads.
In my experience, there are a few things that you need to keep in mind when using this strategy:
Strive to return the correct HTTP response code: HTTP is a ubiquitous protocol and is no doubt understood by your web container. Its response codes fit naturally into REST web services. As such, leverage it! If your web service encounters an error condition, you should do your best to return the correct HTTP status code in whose context, the API error code has meaning. One my biggest headaches in debugging issues with web services occur when developers just unconditionally throw arbitrary (usually runtime) exceptions back up the stack. The result is that everything gets returned back to the caller as an HTTP 500 (Internal Server Error) status code even when that's not the case (e.g. the client sends garbage data and the server just can't process it. Some common HTTP status codes you might want to design for include:
400 Bad Request: There is an issue with the client's request. Note this error isn't just used for things like broken JSON syntax in a POST request, but it is also a legitimate response code for semantic issues as well (i.e. the JSON request payload conformed to the prescribed schema, but there was an issue with the data in the payload, such as a number being negative when it is supposed to be only positive).
401 Unauthorized: The caller's credentials were invalid (i.e. authorization error).
403 Forbidden: The caller's credentials were valid, but their access level isn't sufficient to access the resource (i.e. authentication error).
404 Not Found: The resource of the URL doesn't exist.
500 Internal Server Error: Something bad happened inside the server itself, this error could be anything.
502 Bad Gateway: An error occurred when calling downstream service.
503 Service Unavailable: A useful response code for when you get hammered with a ton of "happy" customers who are inadvertently DDOS'ing your service.
504 Gateway Timeout: Like the 502 status code, but indicates a timeout instead of an actual error with the downstream service, per se.
HTTP response codes are the top-level codes, and API error codes only have meaning within that context: By this, I mean that your API error codes are only meaningful for certain HTTP response codes. For example, in the table of SES error codes, each error code is only tied to a single HTTP(S) response code. The error codes ConfigurationSetDoesNotExist and InvalidParameterValue only make sense when a 400 Bad Request is returned by SES - it wouldn't make sense to return these status codes when a 500 Internal Server Error is returned. Similarly, if you were writing a web service that called downstream services and databases, you might have a FooDownstreamServiceTimedOut error code that you would return with a 504 Gateway Timeout HTTP status code when a downstream web service call timed out to the "Foo" web service. You might also have a MyDatabaseError error code that you would return with a 500 Internal Server Error HTTP status code when your query to the internal DB fails.
Have a uniform error code schema irrespective of status codes: Your clients need to be able to process your error content programmatically. As such, it needs to conform to a certain schema. Ideally, your API error code schema should include the error code (i.e. name or ID, etc.). You also probably want to include a natural language description of the error code and the ID/GUID of the request that you are responding to. For an example of an error schema, see this sample AWS SES response and schema. Additionally, you might also want to consider returning a client ID in the response. This is as much for your own benefit as the client's since it can help you drill down into the data to see if one particular client is getting a glut of particular errors vs. your other clients.
Consider returning natural language descriptions of the error codes in the response: To make things easier on your clients, you might want to consider not just returning the error code in the error payload, but a natural language description as well. This kind of behavior can immediately help confused and busy engineers who really don't care that much about your service quickly diagnose what's happening so that they can resolve the issue ASAP. btw, enabling engineers to quickly diagnose issues with your service increases the all-important "uptime" metric that your customers and managers will no doubt care about.
Don't feel obliged to use integers, use enumerations instead: The notion of "error codes" conjures up images of outdated technologies and codebooks where you had to look up what an error meant. It arose from the programming dark ages when engineers needed to fit all possible errors into a byte of space, or a nibble or whatever. Those days are gone, and your error code can be a string, likely without any meaningful impact on performance. You might as well take advantage and make the error code meaningful, as a means of keeping things simple.
Return info to clients that they might need to debug, but be mindful of security: If possible, return whatever debug info your clients may need. However, if your service potentially deals with sensitive information such as credit card numbers and the like, you probably don't want to pass that info around for obvious reasons.
Hope that helps.
A recommendation by the IETF (internet standards body) is using the application/problem+json mediatype.
Notable is that they don't use random numbers, they use strings (specifically uris) to identify errors.
This is a subjective question, but even if you don't use their format, I'd argue that username-not-provided is better in almost every way to 100001.
I would say this heavily depends on what kind of API you're providing.
I were to always include a field called ack or something similar in every response that has three states: failure, warning, success. Success obviously being everything went well. On warning, the request went through and the JSON will contain the expected output, but it will also include a warning string, or even better in case multiple warnings could occur an array called errors which consists of multiple objects containg code, string and type. This array will also be returned in case of failure, and nothing else but this array.
The array contains one object per error or warning, having a code (I would suggest going with your initial idea of 10001, 10002, ...) and a string explaining the error in a very short phrase (e.g. Username contains invalid characters). The type is either error or warning, which is useful in case of a failure ack that contains not only errors but also warnings.
This makes it easy to look up errors by their code (I would provide a page, also with an API, that contains all the error codes in a table along with their short and long description plus common causes/fixes/etc. - All this information should also be available via an API where they can be accessed by providing the error code) while still having a quick short text response so the user can tell what's wrong in most cases without having to look up the error.
This also allows for easy output of warnings and errors to the end user, not just the developers. Using my idea with the API call to get informations about an error, developers using your API could easily provide full information about errors to end-users when needed (including causes/fixes/whatever you see fit).
Instead of writing your own API standard from scratch adopt one of the already available, for example the JSON API standard:
If you’ve ever argued with your team about the way your JSON responses should be formatted, JSON API can be your anti-bikeshedding tool.
By following shared conventions, you can increase productivity, take advantage of generalized tooling, and focus on what matters: your application.
Clients built around JSON API are able to take advantage of its features around efficiently caching responses, sometimes eliminating network requests entirely.
If you decide to go with JSON API it has a section dedicated to errors and a few error examples.
For many years, many developent companies have created things like bitmask for errors, so they can encode multiple variables inside the error:
000 - all ok
001 - something failed with X
010 - something failed with Y
011 - something failed with X and Y
100 - something failed with Z
101 - something failed with X and Z
The limitation is that that limits the error space into however many bytes you decide on the encoding, like 16 or 32 possible combinations, it may be enough for you, or not.
You see this being common in COM+
https://learn.microsoft.com/en-us/windows/desktop/com/com-error-codes-1
I hope this helps.
We're using the Microsoft Graph .NET Client Library to send requests to the Excel API in order to read or write to Excel files in Office365. We have noticed that the data that we get back from the API is sometimes stale.
For instance, if we add a row to an Excel file, and then immediately read all rows from the same file, even if the add request succeeds, the row will still be missing from the data that we read back. If we wait for a few seconds, the row will show up. This problem does not reproduce consistently, and the delay time varies from less than a second to sometimes tens of seconds. The same problem occurs in update or delete operations as well.
Based on this, we speculate that behind the API, data takes a significant amount of time to propagate across all of Microsoft's servers, and if our requests are not always routed to the same server, we will occasionally hit a server that does not have the latest data.
Could someone who is working on either the Microsoft Graph API or the Excel API verify this guess? We understand that as Microsoft transitions from shipping packaged software to building cloud services, there will be problems and challenges, so we don't expect an immediate solution. However, since our business depends greatly on this API, if there is a known problem, please let us know so that at least we can try to find a workaround on our end.
Any response would be greatly appreciated. Thank you in advance.
Please check
https://dev.office.com/blogs/power-your-apps-with-the-new-excel-rest-api
Copied from the above URL:
Note: Any request that modifies the workbook should be performed in a
persisted session. Find more details on how to create a persisted
session in our documentation.
Create a persisted session
POST .../workbook/CreateSession
content-type: Application/Json
authorization: Bearer {access-token}
{ "persistChanges": true }
Response
HTTP code: 201, Created
content-type: application/json;odata.metadata
{ "#odata.context": "https://graph.microsoft.com/v1.0/$metadata#microsoft.graph.sessionInfo", "id": "{session-id}", "persistChanges": true}
Usage The session ID returned from the CreateSession call is then
passed as a header on subsequent API requests using the
workbook-session-id HTTP header.
GET .../workbook/Worksheets
authorization: Bearer {access-token}
workbook-session-id: {session-id}
I have been using Mandrill webhooks from a long time and till now I haven't encountered this error.
But now I see this error, I am not sure what has caused this ?
Please let me know why this might be happening and what might be the possible solution for the same.
Is it related to my server handling capacity because I have checked for that as well and Mandrill doesnt have too many concurrent request that it is sending to my Apache server, so according to me that is not an issue and also mysql also doesn't seem to be causing the bottleneck, but then I I have not used any benchmarking tool to determine the same.
Please let me know the solution if you guys have encountered something like this.
It seems that the URL is not responding to the request. There could be a few reasons:
If the URL points to an internal server, a firewall could be blocking it or a port number (if given).
Once set up the webhook will send via a POST HTTP verb, however for testing it sends a HEAD request. Quite often web servers (e.g. IIS) will limit what verbs they respond to and will only respond to GET and POST requests.
If that's working your URL should respond with just headers only to acknowledge the request. (HEAD doesn't allow any page content to be sent) so it should only do something like this for a HEAD request:
<?php header( 'Content-Type:' ); // returning 200 ?>
More details on their site
http://help.mandrill.com/entries/22024856-Why-can-t-my-webhook-or-inbound-route-URL-be-verified-
You may wish to try this tool to see what HTTP header result is being returned (if any) or if another error is being returned, just remember that if the URL is internal, it could be blocked to the outside world.
https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en
New here. I've searched quite a bit for a working solution to my problem, but even though I have found posts with promising titles, none of the solutions have worked.
I am deploying an MVC2 web app to a client's server.
I did my development on Win2k8 Server, but they are running Win2k3 sever.
The app's only purpose is to receive some record ID information as HTTP parameters, check in the database for the status of the given record or records, and then return the status information as a simple string such as "Completed" or "Incomplete" in JSON format.
This getJSON code works fine in the development environment.
Inexplicably to me, on the client's server, the getJSON request receives a null response from the application.
There is no cross-domain action AFAIK... the result is the same from the client's server or from my machine via VPN.
In the MVC model's Json code, a common solution for people is to add the "JsonRequestBehavior.AllowGet" attribute to the Json result being returned. I did this long before trying to deploy it, and as I said, it has worked fine in the dev environment.
Using Firebug, I have watched the same request URL get sent to both my local server and the client server - the response headers from both servers are the same, but the response content from my server is shown as:
{"Result":"No Data"}
Which is what I want.
There is literally no content shown in the response from the client's server..? But the request gets an HTTP 200 code and is recorded as a success in the reponse's status attribute.
The response header content type in both situations is "application/json"
But wait, there is more!
If I manually enter the request to each server in the Firefox nav bar, and hit enter, in both cases it responds with:
{"Result":"No Data"}
Which is what I want. So why can I get the result I want from the MVC app on the client's server only when I hand-enter the request URL in Firefox, but not from the Javascript code?
I have tried forcing different output content types ... using the jQuery ajaxSetup method...
$.ajaxSetup({
async: false,
dataType: 'text'
});
or
$.ajaxSetup({
async: false,
dataType: 'html'
});
and again wtih 'script', and 'json'. I also tried the conversion options such as 'text json' or 'html json' or 'json text' and so forth.
Some of the posts I'm reading, and my gut feeling, though, suggest the problem is not the jQuery code making the request that is at fault... I don't see how the same jQuery request point to a different server running the same app would suddenly cause that server to send back a 'null' value.
By null, I want to be clear... I mean nothing is sent. There is no {} or {null} or any sign of JSON... just blank whiteness of non-existence :P
Even if nobody knows the answer, I would love some input perhaps suggesting where I should focus my sleuthing ... client or server?
If the problem is the server, it seems hard to really know that the MVC stuff is running 100% on the IIS6 server, but in general it seems to work. I have a different MVC app running on the client server which responds to the virtual paths, and generally runs the same as on dev machine.
I have found one thing ... the request headers are somewhat different? The Request Headers sent to the IIS7 setup include an "X-Requested-With: XMLHttpRequest", "referrer" , and "cookie" field/value.
I could guess that the lack of the "X-requested-with: XMLHttpRequest" in the IIS6 request headers is a clue, but I do not see then how the same javascript code pointing at a different server can generate different request headers itself. So how else are those being generated?
The javascript is embedded in an ASP.NET page, btw.
Oooh.. frustration!
Thanks for any input.
Odd Progress ... apparently there is some sort of issue with IIS6 handling the query. Although I have not payed any attention to JSONP, a post elsewhere suggested that sometimes use the "&callback=?" parameter at the end of a .getJSON request URL would force it into GET mode and that worked frequently for problems getting data from the server. So I did that... and it did work, sort of. The proper {"Result":"No Data"} was returned in response to the request... which seems good. However, the way that the JSONP callback works, it generates its own script to do the calling and fetching and interpreting of the incoming JSON. In this case, it interprets the JSON to need a label which it does not have, thus an error is thrown "invalid label" ... there must be some way to hack things to just deliver the JSON, but the whole required use of JSONP callbacks suggests that the server configuration is wrong, right? Or why does it work without JSONP for IIS7 and not IIS6?
Despite my not liking the callback JSONP solution, it appears to work ok. The error is still returned about an invalid label, but that does not appear to stop the remaining javascript from running... and so the application is working with IIS6 now. I have not tested the fix of using the callbacks and JSONP against IIS7 yet, but I expect it will work well enough.
Here is a link to the discussion that lead me to my current solution. I do still hope to find a more elegant solution, however.
NeoWin.net
Are you certain that your App Extension Mappings are set up correct?
Follow this article for running MVC2 on IIS6 and ensure all the different configurations have been done, that's probably the first step before going further and investigating specifics.
I'm really inclined to believe it's related to HTTP Verbs.