I'm trying to set up an inbound policy on an API in Azure API Manager that validates the JSON in POST request body before passing it on to the backend.
Should I be using a JSON schema and validate against that (how?) or if I should write my own code, using context.Request.Body to inspect each field in the request body or is it just plain wrong to try to validate a request body in APIM, should that be left to the backend?
We are currently validating in the API operations policy with a <set-body> to validate (and re-shape body if required). In this policy we're collecting all errors which we then throw with an exception:
<set-body>#{
var body = context.Request.Body.As<JObject>(true);
string businessSystemID = context.Request.Headers.GetValueOrDefault("SenderBusinessSystemID", "");
var result = new JObject();
string cleanUp = body.ToString().Replace("\r\n","");
var root = JObject.Parse(cleanUp);
var valid = false;
var errors = new JArray();
var returnValue = new JObject();
returnValue["errors"] = errors;
if(root["CostAllocation"] != null)
{
if(businessSystemID != string.Empty)
{
root["CostAllocation"]["SenderBusinessSystemID"] = businessSystemID;
}
if(root["CostAllocation"]["ReferenceID"] != null)
{
var referenceIDValidator = #"^[A-Z0-9]{0,35}$";
var referenceID = root["CostAllocation"]["ReferenceID"];
valid = new Regex(referenceIDValidator, RegexOptions.IgnoreCase).Match(referenceID.ToString()).Success;
if(!valid)
{
var error = new JObject();
error["property"] = "ReferenceID";
error["validator"] = referenceIDValidator;
error["value"] = referenceID;
error["message"] = "No valid 'ReferencedId'";
errors.Add(error);
}
}
if(root["CostAllocation"]["UserID"] != null)
{
var userIDValidator = #"^[\w]{4,12}$";
var userID = root["CostAllocation"]["UserID"];
valid = new Regex(userIDValidator, RegexOptions.IgnoreCase).Match(userID.ToString()).Success;
if(!valid)
{
var error = new JObject();
error["property"] = "UserID";
error["validator"] = userIDValidator;
error["value"] = userID;
error["message"] = "No valid 'UserID'";
errors.Add(error);
}
}
...
if(errors.Count > 0)
{
throw new Exception(returnValue.ToString());
}
return root.ToString(Newtonsoft.Json.Formatting.None);
}</set-body>
Exception is caught in <on-error>, handled and formatted:
<on-error>
<choose>
<when condition="#(context.LastError.Message.Contains("Expression evaluation failed")==true)">
<set-status code="400" reason="Bad request" />
</when>
<otherwise>
<set-status code="500" reason="Error" />
</otherwise>
</choose>
<set-body template="none">#{
return context.LastError.Message.Replace("Expression evaluation failed.","").Trim();
}</set-body>
<base />
</on-error>
We're aware that coding effort is high with this approach and would rather prefer for JSON schema checking natively supported directly in API Management. Having schema checking handled in a back-end and with that adding latency by passing on the request is not an option for us.
One option which I'm currently thinking of is parsing JSON schema in the API CI/CD process, generating a generic validation & replacing it in the API operations policy.
Validating payload is perfectly fine. Just be aware that doing so will require caching whole request body on APIM side which is not done by default. JSON schema validator is not available in policy expressions, so you have to resort to manual validation.
Related
I'm facing a behavior in Minimal API that I can't understand.Consider the following simple Minimal API:
var builder = WebApplication.CreateBuilder(args);
var app = builder.Build();
app.UseExceptionHandler((exceptionApp) =>
{
exceptionApp.Run(async context =>
{
context.Response.ContentType = MediaTypeNames.Text.Plain;
var feature = context.Features.Get<IExceptionHandlerPathFeature>();
if (feature?.Error is BadHttpRequestException ex)
{
context.Response.StatusCode = 400;
var message =
(ex.InnerException is JsonException)
? "The request body is an invalid JSON"
: "Bad Request";
await context.Response.WriteAsync(message);
}
else
{
context.Response.StatusCode = 500;
await context.Response.WriteAsync("There is a problem occured");
}
});
});
app.MapPost("/models", (Model model) => Results.Created(string.Empty, model));
app.Run();
public record Model(int Value, string Title);
When I run the application in the Development environment, and pass an invalid JSON like
{
"value": 1,
"Title": Model #1
}
the custom exception handler is called and I have to control the behavior of the API. But whenever
I run the application in the Production environment, the framework automatically returns a
"bad request" response without letting me control the response.
Could anyone explain this behavior to me? I really need my exception handler to handle invalid input
JSON exceptions.
Thanks
After digging into ASP.Net Core source code for a while, I found that the following line resolves the issue.
builder.Services.Configure<RouteHandlerOptions>(o => o.ThrowOnBadRequest = true);
I have an APIM policy to send a request and response to an Azure Event Hub:
<outbound>
<log-to-eventhub logger-id="my-logger">#{
var request = context.Request.Body.As<string>(preserveContent: true);
var response = context.Response.Body?.As<string>(preserveContent: true);
var json = new JObject(
new JProperty("DateTimeStamp", DateTime.UtcNow),
new JProperty("IpAddress",context.Request.IpAddress),
new JProperty("RequestBody", request),
new JProperty("ResponseBody", response),
new JProperty("ResponseCode", context.Response.StatusCode)
);
return json.ToString();
}</log-to-eventhub>
<base />
</outbound>
A request made via Postman or any other client shows the response has the correct body but in the code above context.Response.Body is always null, hence the null check. The message shows up in the event hub fine with a correct request message but null response.
There are other policies on the API but this is the only outbound policy. What could be causing this?
Edit with full Trace message taking out the null check and using JObject as suggested.
log-to-eventhub (0.405 ms)
{
"messages": [
{
"message": "Expression evaluation failed.",
"expression": "\n var request = context.Request.Body.As<string>(preserveContent: true);\n var response = context.Response.Body.As<JObject>(preserveContent: true);\n\n var json = new JObject(\n new JProperty(\"DateTimeStamp\", DateTime.UtcNow),\n new JProperty(\"ServiceName\",context.Deployment.ServiceName),\n new JProperty(\"RequestId\",context.RequestId),\n new JProperty(\"IpAddress\",context.Request.IpAddress),\n new JProperty(\"OperationName\",context.Operation.Name),\n new JProperty(\"RequestBody\", request),\n new JProperty(\"ResponseBody\", response.ToString()),\n new JProperty(\"ResponseCode\", context.Response.StatusCode)\n );\n \n return json.ToString();\n ",
"details": "Object reference not set to an instance of an object."
},
"Expression evaluation failed. Object reference not set to an instance of an object.",
"Object reference not set to an instance of an object."
]
}
In place of As<string> try using As<JObject> which would return json JObject.
var response = context.Response.Body.As<JObject>(preserveContent: true);
JObject() Initializes a new instance of the JObject class.
for more information check https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_Linq_JObject.htm
I'm trying to code a middleman API that logs calls and other details from internal users to an external API.
When I try to POST to the external API from my Controller, I get 415 unsupported media type.
I set up my client in the controller constructor like this:
client.BaseAddress = new Uri("https://restapi.***.com/customers/");
client.DefaultRequestHeaders.Clear();
client.DefaultRequestHeaders.Accept.Add(new System.Net.Http.Headers.MediaTypeWithQualityHeaderValue("application/json"));
client.DefaultRequestHeaders.Add("X-AppSecretToken", Auth.APPSECRETTOKEN);
client.DefaultRequestHeaders.Add("X-AgreementGrantToken", Auth.AGREEMENTGRANTTOKEN);
My POST method looks like this:
var json = JsonConvert.SerializeObject(customer, Formatting.Indented);
using (var stringContent = new StringContent(json))
{
stringContent.Headers.ContentType.CharSet = "";
HttpResponseMessage response = await client.PostAsync(client.BaseAddress, stringContent);
if (!response.IsSuccessStatusCode)
{
return StatusCode((int)response.StatusCode);
}
}
return CreatedAtAction("GetCustomer", new { id = customer.ID }, customer);
I've been looking around and found a lot of comments telling me to use Stringcontent, but I also found a couple of responses saying ByteArrayContent - none of them work.
Can anyone help me?
EDIT: When I run the code with breakpoints it seems like some of the properties in the incoming customer object are set even though I didn't set them in my Postman call.
Example; the external API returns a customernumber when I give it the 5 properties that are obligatory. But when I call my internal API from Postman, sending only those 5 obligatory properties, it autopopulates the customernumber with a 0.
Could this be the source of the error? and how do I tell .net core to not autopopulate the customernumber?
EDIT2: I changed my stringContent to include encoding and used a different overload, so the using line now says
using (var stringContent = new StringContent(json, Encoding.UTF8, "application/json"))
And I removed
stringContent.Headers.ContentType.Charset = "";
to reflect the fact that I tried setting the encoding.
The return code changed from 415 to 400 Bad Request when I changed that.
EDIT3:
Tried NOT serializing with Json.Net, and instead used JObjects and Jproperties;
public async Task<ActionResult<Customer>> PostCustomer([FromBody]Customer customer)
{
JObject payload = new JObject(
new JProperty("currency", customer.Currency),
new JProperty("name", customer.Name),
new JProperty("customergroup",
new JObject(new JProperty("customergroupNumber",
customer.CustomerGroup.CustomerGroupNumber)
)),
new JProperty("paymentTerms",
new JObject(new JProperty("paymentTermsNumber",
customer.PaymentTerms.PaymentTermsNumber)
)),
new JProperty("vatZone",
new JObject(new JProperty("vatZoneNumber",
customer.VatZone.VatZoneNumber)
))
);
using (var stringContent = new StringContent(payload.ToString(), Encoding.UTF8, "application/json"))
{
HttpResponseMessage response = await client.PostAsync(client.BaseAddress, stringContent);
if (!response.IsSuccessStatusCode)
{
return StatusCode((int)response.StatusCode);
}
}
return CreatedAtAction("GetCustomer", new { id = customer.CustomerNumber }, customer);
}
Still 400 Bad Request
This is a case of capitalizing - simple really.
My POST request JSON had an object named customergroup - changed it to customerGroup, and it worked.
I am using the Salesforce REST API. I have created a new custom object within SFDC and want to POST data to it. I can happily POST JSON documents to the standard SFDC objects such as Account or Contact. However, when I try posting to my custom object I receive a timeout from SFDC.
Here is the method I am using to perform all POSTS (whether standard or custom)
public static BsonDocument PostJSONToSFDC(string uri, BsonDocument postDoc, string method, HPSUtilities.Transformation.Transformation transformation)
{
string accessToken = transformation.sfdcAccess.accessToken;
string instanceUri = transformation.sfdcAccess.instanceUri;
string uri2 = instanceUri + uri;
System.Net.WebRequest req = System.Net.WebRequest.Create(uri2);
req.ContentType = "application/json";
req.Method = method;// "POST" or "PATCH"; // a PATCH alternative uses POST with url parm="?_HttpMethod=PATCH"
req.Headers.Add("Authorization: Bearer " + accessToken);
string postDocAsString = postDoc.ToJson();
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(postDocAsString);
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream();
os.Write(bytes, 0, bytes.Length); //Push it out there
os.Close();
System.Net.WebResponse resp = req.GetResponse();
if (resp == null) return null;
System.IO.StreamReader sr =
new System.IO.StreamReader(resp.GetResponseStream());
string rs = sr.ReadToEnd().Trim();
MongoDB.Bson.BsonDocument doc2;
if (rs.Equals("") && method.Equals("PATCH", StringComparison.CurrentCultureIgnoreCase))
{
// For successful PATCHs (updates), SFDC mysteriously returns a completely empty response.
// In this case let's create something more meaningful!
doc2 = new BsonDocument();
doc2.Add("success", "true");
}
else
{
doc2 = MongoDB.Bson.Serialization.BsonSerializer.Deserialize<MongoDB.Bson.BsonDocument>(rs);
}
return doc2;
}
Given that this code works when POSTing or PATCHing to SFDC I believe that I must be hitting some sort of SFDC restriction for custom objects. Is it necessary to mark SFDC custom objects as API-enabled? Is anyone aware of any other issues that might be causing my timeout?
I am using ServiceStack to create a C# client to a JSON RESTful service. I have this code that returns my DTO:
Search result = restClient.Get (search);
This works fine, but in order to effectively debug the search results coming back I need to output the text content from the underlying HTTP Response object. (I don't know all the elements in the response yet in order to add them to the DTO).
Is there any way I can get hold of the underlying HTTP response, and thus the full text content, from my result object?
Thanks in advance.
#adamfowleruk
When inheriting from ServiceStack's built-in Service you can access the underlying Request and Response directly from the Response class with:
public class MyService : Service
{
public object Get(Request request)
{
base.Request ...
base.Response ...
}
}
You won't see the response output in your service or filters since it writes directly to the response stream and is the last thing that ServiceStack does after executing your service and all response filters.
For diagnosing HTTP I recommend using Fiddler or WebInspector also ServiceStack's built-in Request Logger might help as well.
Consuming a ServiceStack service
If you're using the C# Service Clients you can simply ask for what you want, e.g. you can access the returned response as a raw string:
string responseJson = client.Get<string>("/poco/World");
Or as raw bytes:
byte[] responseBytes = client.Get<byte[]>("/poco/World");
Or as a Stream:
using (Stream responseStream = client.Get<Stream>("/poco/World")) {
var dto = responseStream.ReadFully().FromUtf8Bytes().FromJson<PocoResponse>();
}
Or even access the populated HttpWebResponse object:
HttpWebResponse webResponse = client.Get<HttpWebResponse>("/poco/World");
webResponse.Headers["X-Response"] //World
using (webResponse)
using (var stream = webResponse.GetResponseStream())
using (var sr = new StreamReader(stream)) {
string response = sr.ReadToEnd();
}
You can also introspect the HttpWebResponse by using Global and Local Response filters, e.g:
JsonServiceClient.HttpWebResponseFilter = httpRes => { .. };
Or using a Local filter:
var client = new JsonServiceClient(baseUrl) {
ResponseFilter = httpRes => { .. }
};
Consuming a 3rd Party Service
If you're consuming a 3rd Party REST/HTTP API you can use a responseFilter: in ServiceStack's HTTP Util extensions:
List<GithubRepo> repos = "https://api.github.com/users/{0}/repos".Fmt(user)
.GetJsonFromUrl(responseFilter: httpRes => {
var remaining = httpRes.Headers["X-Api-Remaining"];
})
.FromJson<List<GithubRepo>>();
I use Fiddler to debug my services. It gives you all sorts of cool HTTP debugging facilities.
http://www.fiddler2.com/fiddler2/
I like to use RestConsole. It is a Chrome Extension and you can easily submit POST requests and see the response. It is also handy to create sample data and then step into the ServiceStack code and see what's happening. The ServiceStack PluralSight course has a nice demo of how to use them together.
Thanks to the above help I found the right answer. Documenting here for others:-
SearchResponse result = null; // my ServiceStack DTO
HttpWebResponse webResponse = restClient.Get<HttpWebResponse>(
completePath("/v1/search",qp)); // builds the URL with parameters
using (var stream = webResponse.GetResponseStream())
using (var sr = new StreamReader(stream)) {
var text = sr.ReadToEnd();
log.log ("response text: " + text); // *** PRINTING STRING VALUE HERE FOR DEBUG
result = text.FromJson<SearchResponse>();
}
// Now do something useful with the result DTO object
log.log ("RESULT: " + result.ToString ());
for (int i = 0; i < result.Results.Length; i++) {
log.log ("Result " + i + ": " + result.Results[i].ToString());
}