I have a method for parsing a config file into a dictionary. I'm unsure how it should behave if a config parameter is missing. Should it use a default value and log an error or raise an exception?
I suggest that you separate: (1) the parsing of the configuration file and storing the parsed details into a dictionary, from (2) retrieving name=value pairs from the dictionary. This separation of concerns will then enable you to provide an overloaded API for (2) that specifies if a missing name=value pair should result in a default value being returned or an exception being raised. For example (pseudo code):
cfg = parseConfigurationFile("example.cfg")
x = cfg.lookupString("x"); // throws an exception if the name=value is missing
y = cfg.lookupString("y", "hello, World!"); // returns default value if name=value is missing
I also suggest that the API should provide type-safe lookup methods such as lookupInt(), lookupBoolean(), lookupDouble() and so on. Those methods should throw an exception if a looked-up value cannot be parsed into the specified type.
Edit to respond to a comment
"Thanks for the example. I was more wondering if it was a good idea to even
provide default settings and start the application if the config was wrong."
I like the Fail Fast Principle, so I recommend that if any configuration data is invalid, your application should report an error and stop, rather than try to silently repair the error (perhaps by using a default value instead of a bad configuration value) and continue on.
However, I don't think you should necessarily view missing name=value pairs as being an error. Instead, it is valid to use a default value for a missing value. If you take this to an extreme by allowing all the configuration name=value pairs to be optional, then your application will be able to work "out of the box" without any configuration file at all, which arguably improves the application's ease of use for new users.
A few years ago, I wrote Config4* (C++ and Java libraries for parsing a particular configuration-file syntax). Config4* provides an elegant way to enable any/all name=value pairs to be optional: it's what the Config4* manual calls fallback configuration. If you want to learn about that, then I suggest you skim-read Chapter 2 of the Config4* Getting Started Guide to get an understanding of the configuration syntax, and then read Chapter 3 of the same manual to understand the API. Pay particular attention to Sections 3.6.2 (Parsing Embedded Configuration) and 3.6.3 (Using Fallback Configuration).
Related
We are generating an OpenApi definition using Swagger/Swashbuckle. This definition is then imported into Azure API Management.
We have some querystring parameters on get requests that we have marked as required. Our validation ensures the querystring parameters are present and valid, otherwise we return a 400 Bad Request with details of which parameters are invalid/missing. The relevant part of the OpenAPI definition is below. Two querystring parameters (marked as required) and one path parameter (marked as required).
My problem is the way the OpenApi definition is converted into APIM operations.
The required querystring parameters are added as template parameters and they are added to the operation url. This means if they are not provided, APIM cannot match the request to an operation and we return a 404 to the caller rather than the helpful 400 that the backend would return.
I can't add easily add empty values into the querystring. I can't do that in the inbound policy of the operation as it doesn't match the operation. Doing it in the global inbound policy would mean I had to identify the operation myself (this is just one of many). Similarly, while I can return a 400 bad request in the onerror policy, I can't easily tell the caller what was wrong with the request.
I think it's built into the import process. When I changed the template parameters to query parameters in the portal and exported, the OpenApi definition was practically identical. When I reimported the exported template, the same thing happened. I also tried going via Wadl which looked more promising but I couldn't reimport that template.
Is there any way to move template query string parameters to be query string parameters? Any other options?
At the moment (since 2018) there is the bug in Azure APIM API import. Link.
It's status under review. We tried to raise this directly to Microsoft, but there was no solution provided from their side.
I have read a few posts here and on GitHub regarding .NET Core 3.1's change in tightening of the allowance of serializing data. My actual exception is:
System.Text.Json.JsonException: 'A possible object cycle was detected which is not supported. This can either be due to a cycle or if the object depth is larger than the maximum allowed depth of 0.'
When I search the web, there are very few results, but from what I gather, it's telling me that it doesn't like the related data that is being serialized. Well, in the past, this hasn't been a problem. After reading this post, it says to install the Newtonsoft Json serializer package. I did that, and added the verbiage in Startup, but got the same result. So, I read another post here where the member who answered the question returned a JsonResult in the method. I tried this as a test and it worked just fine. Problem is, I need to return the data (serialized) to a view.
My question is, why can the standard System.Text.Json.JsonSerializer not serialize my data, yet a JsonResult can? Either way, I just need to serialize my data and cannot get past this error and any help would be greatly appreciated. If I have to return the data via an API and continue to use the JsonResult, I'm fine with that, but am concerned as to why it works.
As #poke suggested, using Json.Serialize works. I was using System.Text.Json.JsonSerializer.Serialize(Model) and for some strange reason, this throws the same exception even though the Ignore option is set in Startup. More research is required, but at least this gives me something to look into.
What are the good choice for API error code response pattern?
Instead of using different codes indicating different type of error
100001 // username not provided
100002 // password not provided
100003 // password too short
...
I see some other use patterns like the following (non-sequential) ...
20000
20001
20004
20015
Are there any other recommendations?
In my experience developing and using web services, I have found that a strategy of using a combination of top-level HTTP status codes and lower level API error codes work reasonably well. Note that the lower level API error codes don't need to be integers, but can be any enumeration. For a well-known public example, AWS Simple Email Service (SES) uses this strategy of using both HTTP status codes and API level error codes. You can see a sample error code response for SES here. Note that although SES uses XML response error payloads, this strategy works equally well for JSON response payloads.
In my experience, there are a few things that you need to keep in mind when using this strategy:
Strive to return the correct HTTP response code: HTTP is a ubiquitous protocol and is no doubt understood by your web container. Its response codes fit naturally into REST web services. As such, leverage it! If your web service encounters an error condition, you should do your best to return the correct HTTP status code in whose context, the API error code has meaning. One my biggest headaches in debugging issues with web services occur when developers just unconditionally throw arbitrary (usually runtime) exceptions back up the stack. The result is that everything gets returned back to the caller as an HTTP 500 (Internal Server Error) status code even when that's not the case (e.g. the client sends garbage data and the server just can't process it. Some common HTTP status codes you might want to design for include:
400 Bad Request: There is an issue with the client's request. Note this error isn't just used for things like broken JSON syntax in a POST request, but it is also a legitimate response code for semantic issues as well (i.e. the JSON request payload conformed to the prescribed schema, but there was an issue with the data in the payload, such as a number being negative when it is supposed to be only positive).
401 Unauthorized: The caller's credentials were invalid (i.e. authorization error).
403 Forbidden: The caller's credentials were valid, but their access level isn't sufficient to access the resource (i.e. authentication error).
404 Not Found: The resource of the URL doesn't exist.
500 Internal Server Error: Something bad happened inside the server itself, this error could be anything.
502 Bad Gateway: An error occurred when calling downstream service.
503 Service Unavailable: A useful response code for when you get hammered with a ton of "happy" customers who are inadvertently DDOS'ing your service.
504 Gateway Timeout: Like the 502 status code, but indicates a timeout instead of an actual error with the downstream service, per se.
HTTP response codes are the top-level codes, and API error codes only have meaning within that context: By this, I mean that your API error codes are only meaningful for certain HTTP response codes. For example, in the table of SES error codes, each error code is only tied to a single HTTP(S) response code. The error codes ConfigurationSetDoesNotExist and InvalidParameterValue only make sense when a 400 Bad Request is returned by SES - it wouldn't make sense to return these status codes when a 500 Internal Server Error is returned. Similarly, if you were writing a web service that called downstream services and databases, you might have a FooDownstreamServiceTimedOut error code that you would return with a 504 Gateway Timeout HTTP status code when a downstream web service call timed out to the "Foo" web service. You might also have a MyDatabaseError error code that you would return with a 500 Internal Server Error HTTP status code when your query to the internal DB fails.
Have a uniform error code schema irrespective of status codes: Your clients need to be able to process your error content programmatically. As such, it needs to conform to a certain schema. Ideally, your API error code schema should include the error code (i.e. name or ID, etc.). You also probably want to include a natural language description of the error code and the ID/GUID of the request that you are responding to. For an example of an error schema, see this sample AWS SES response and schema. Additionally, you might also want to consider returning a client ID in the response. This is as much for your own benefit as the client's since it can help you drill down into the data to see if one particular client is getting a glut of particular errors vs. your other clients.
Consider returning natural language descriptions of the error codes in the response: To make things easier on your clients, you might want to consider not just returning the error code in the error payload, but a natural language description as well. This kind of behavior can immediately help confused and busy engineers who really don't care that much about your service quickly diagnose what's happening so that they can resolve the issue ASAP. btw, enabling engineers to quickly diagnose issues with your service increases the all-important "uptime" metric that your customers and managers will no doubt care about.
Don't feel obliged to use integers, use enumerations instead: The notion of "error codes" conjures up images of outdated technologies and codebooks where you had to look up what an error meant. It arose from the programming dark ages when engineers needed to fit all possible errors into a byte of space, or a nibble or whatever. Those days are gone, and your error code can be a string, likely without any meaningful impact on performance. You might as well take advantage and make the error code meaningful, as a means of keeping things simple.
Return info to clients that they might need to debug, but be mindful of security: If possible, return whatever debug info your clients may need. However, if your service potentially deals with sensitive information such as credit card numbers and the like, you probably don't want to pass that info around for obvious reasons.
Hope that helps.
A recommendation by the IETF (internet standards body) is using the application/problem+json mediatype.
Notable is that they don't use random numbers, they use strings (specifically uris) to identify errors.
This is a subjective question, but even if you don't use their format, I'd argue that username-not-provided is better in almost every way to 100001.
I would say this heavily depends on what kind of API you're providing.
I were to always include a field called ack or something similar in every response that has three states: failure, warning, success. Success obviously being everything went well. On warning, the request went through and the JSON will contain the expected output, but it will also include a warning string, or even better in case multiple warnings could occur an array called errors which consists of multiple objects containg code, string and type. This array will also be returned in case of failure, and nothing else but this array.
The array contains one object per error or warning, having a code (I would suggest going with your initial idea of 10001, 10002, ...) and a string explaining the error in a very short phrase (e.g. Username contains invalid characters). The type is either error or warning, which is useful in case of a failure ack that contains not only errors but also warnings.
This makes it easy to look up errors by their code (I would provide a page, also with an API, that contains all the error codes in a table along with their short and long description plus common causes/fixes/etc. - All this information should also be available via an API where they can be accessed by providing the error code) while still having a quick short text response so the user can tell what's wrong in most cases without having to look up the error.
This also allows for easy output of warnings and errors to the end user, not just the developers. Using my idea with the API call to get informations about an error, developers using your API could easily provide full information about errors to end-users when needed (including causes/fixes/whatever you see fit).
Instead of writing your own API standard from scratch adopt one of the already available, for example the JSON API standard:
If you’ve ever argued with your team about the way your JSON responses should be formatted, JSON API can be your anti-bikeshedding tool.
By following shared conventions, you can increase productivity, take advantage of generalized tooling, and focus on what matters: your application.
Clients built around JSON API are able to take advantage of its features around efficiently caching responses, sometimes eliminating network requests entirely.
If you decide to go with JSON API it has a section dedicated to errors and a few error examples.
For many years, many developent companies have created things like bitmask for errors, so they can encode multiple variables inside the error:
000 - all ok
001 - something failed with X
010 - something failed with Y
011 - something failed with X and Y
100 - something failed with Z
101 - something failed with X and Z
The limitation is that that limits the error space into however many bytes you decide on the encoding, like 16 or 32 possible combinations, it may be enough for you, or not.
You see this being common in COM+
https://learn.microsoft.com/en-us/windows/desktop/com/com-error-codes-1
I hope this helps.
I'm trying to build a logic app that inserts data into a Sql database. The data is coming from s Stream Analytics job, outputting it on a Service Bus topic, consumed in Logic Apps in Service Bus trigger.
To populate the properties of the row inserted (lets say it only has one column 'Name'), I've found that this should work using following syntax:
"body": {
"Name": "#{json(decodeBase64(triggerBody()['ContentData'])).Name}"
},
Provided the message body contains a 'Name' property.
However I get following error message when running this:
{"code":"InvalidTemplate","message":"Unable to process template language expressions in action 'Insert_row' inputs at line '1' and column '2017': 'The template language function 'json' parameter is not valid. The provided value '#\u0006string\b3http://schemas.microsoft.com/2003/10/Serialization/��{\"time\":\"2016-05-25T10:29:17.4953250Z\",\"Name\":\"Y-Axis\",\"Value\":81.0,\"Date\":\"2016-05-25T10:29:17.4953250\",\"EventProcessedUtcTime\":\"2016-05-25T10:29:17.5525449Z\",\"PartitionId\":2,\"EventEnqueuedUtcTime\":\"2016-05-25T10:29:17.2220000Z\"}\u0001' cannot be parsed: 'Unexpected character encountered while parsing value: #. Path '', line 0, position 0.'. Please see https://aka.ms/logicexpressions#json for usage details.'."}
So it seems like that the content is enclosed in another envelope that is preventing json parsing to work.
1) Any simple way how to get around this?
2) Isn't such an integration all within Microsoft Stack just supposed to work without this mocking around?
Thanks,
Stefan
with the limited string functions available in he workflow definition language I had to use a long winded way for removing the additional strings
#{json(substring(replace(decodeBase64(triggerBody()['ContentData']),'#string3http://schemas.microsoft.com/2003/10/Serialization/��', ''),0,sub(length(replace(decodeBase64(triggerBody()['ContentData']),'#string3http://schemas.microsoft.com/2003/10/Serialization/��', '')),1))).fieldname}
is there a better way to do this
Thanks for reporting this, you're right it should simply work. There is a known issue where ASA ServiceBus output JSON is being wrapped in a XML header. It will be addressed in a near future but can't specify a particular date. Could you please workaround it (maybe using substring/replace) until then ?
cheers,
Chetan
Sorry for the delay in getting back, this issue is now fixed. Its exposed under a new compatibility level (1.1) to avoid breaking existing solutions. Please use this URL to configure your job with compat level 1.1 and give it a try.
cheers!
I have an app that I'm working on converting from CF8 to CF10 and some of my remote CFCs where the data coming back should be JSON are now failing because there seems to be a "//" pre-pended to the returned data. For example here's an output of a returned structure:
//{"SUCCESS":true,"ERRORS":[],"DATA":{"COLUMNS":["AUTHRESULT","SPID","EMAIL","RID"],"DATA":[[true,361541,"user#domain.com",""]]}}
The same function run through the same CFC on the CF8 server gives:
{"ERRORS":[],"SUCCESS":true,"DATA":{"COLUMNS":["AUTHRESULT","SPID","EMAIL","RID"],"DATA":[[true,361541,"user#domain.com",""]]}}
The CFC that proxies all requests does have returnFormat="JSON" - but there is no SerializeJSON() being called in either the proxyCFC or the CFC that is extended from proxyCFC.
I'm not sure what's the best way to handle this. Trimming off the '//' in the response would be possible but it doesn't seem "right". I need to address it on the CF10 end of things because these functions are in use not only in our app, but some remote apps as well (and some are through http:// posts and some are through jQuery Ajax calls).
That is a server side setting in the ColdFusion admin, under settings. Prefix serialized JSON with. It is enabled by default for security. Protects web services, which return JSON data from cross-site scripting attacks by prefixing serialized JSON strings with a custom prefix.. Perhaps you had turned this off on your ColdFusion 8 server. I do not recommend turning it off though.
See this post from Raymond Camden - Handling JSON with prefixes in jQuery and jQueryUI
NOTE: this setting can also be set per-application by setting secureJSON and secureJSONPrefix in your Application.cfc file. See the documentation about that here - Application variables.
secureJSON - A Boolean value that specifies whether to add a security prefix in front of the value that a ColdFusion function returns in JSON-format in response to a remote call.
The default value is the value of the Prefix serialized JSON setting in the Administrator Server Settings > Settings page (which defaults to false). You can override this value in the cffunction tag.
secureJSONPrefix - The security prefix to put in front of the value that a ColdFusion function returns in JSON-format in response to a remote call if the secureJSON setting is true.
The default value is the value of the Prefix serialized JSON setting in the Administrator Server Settings > Settings page (which defaults to //, the JavaScript comment character).