I have some ethernet device which collect data and it's possible to download it via data export interface: HTTP-GET query returns the data in [Content-Type: text/plain Charset: utf-8]
I saw this: How to make an HTTP request from SSIS? - it rather doesn't work for me (C# is a little Chinese for me) and it's about how to fetch this data to variable into SSIS
In your SSIS package add a C# Script Task
Edit the Script Task
At the top with the other using statements add using System.Net;
in Main use the following code snippet to make a GET request (Note: Change "https://somewhere.com/contacts/get" to your actual endpoint.)
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("https://somewhere.com/contacts/get");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
using(HttpWebResponse response = (HttpWebResponse)request.GetResponse())
using(Stream stream = response.GetResponseStream())
using(StreamReader reader = new StreamReader(stream))
{
return reader.ReadToEnd();
}
Related
I have an app that uses WebClient to fetch JSON data from ComicVine as follows:
WebClient client = WebClient.builder()
.baseUrl(url)
.defaultHeaders(
headers -> {
headers.add(HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE);
headers.add(HttpHeaders.USER_AGENT, "ComiXed/0.7");
})
.build();
Mono<ComicVineIssuesQueryResponse> request =
client
.get()
.uri(url)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(ComicVineIssuesQueryResponse.class);
ComicVineIssuesQueryResponse response = request.block();
For a time this worked. But then, all of a sudden, it's throwing the following root exception when it executes:
Caused by: org.springframework.web.reactive.function.UnsupportedMediaTypeException: Content type 'application/json' not supported for bodyType=org.comixed.scrapers.comicvine.model.ComicVineIssuesQueryResponse
at org.springframework.web.reactive.function.BodyExtractors.lambda$readWithMessageReaders$12(BodyExtractors.java:201)
I'm not sure why it all of a sudden won't process JSON data. My unit test, which is explicitly returning JSON data and setting the content type properly:
private MockWebServer comicVineServer;
this.comicVineServer.enqueue(
new MockResponse()
.setBody(TEST_GOOD_BODY)
.addHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE));
Any ideas why this is the case? It's happening across multiple classes that use this same setup for WebClient and for testing.
After doing some digging, I added the following code to get the JSON as a String and then use ObjectMapper to convert it to the target type:
Mono<String> request =
client
.get()
.uri(url)
.accept(MediaType.APPLICATION_JSON)
.retrieve()
.bodyToMono(String.class);
String value = request.block();
ObjectMapper mapper = new ObjectMapper();
ComicVineIssuesQueryResponse response = mapper.readValue(value, ComicVineIssuesQueryResponse.class);
This quickly exposed the underlying problem, which was that two instance variables in the response were annotated with the same JSON field name. Once I fixed that, things started working correctly again.
you can parse the json content to string without calling block method.
option 1) Jackson2Tokenizer
option 2) put your code which is calling "objectMapper.readValue(..) .." inside map operator.
I have a WCF service with a webHttpBinding defined. The interface has a single method:
[OperationContract(Action = "*")]
[WebGet(UriTemplate = "/",RequestFormat = WebMessageFormat.Json, ResponseFormat = WebMessageFormat.Json)]
Stream GetServerInfo();
Which returns a stream with encoded JSON:
WebOperationContext.Current.OutgoingResponse.Headers.Add("Access-Control-Allow-Origin", "*");
ServerData r = ServerData.Instance;
r.upTime = (DateTime.Now - r._startupTime).ToString(#"dd\.hh\:mm\:ss");
using (Process proc = Process.GetCurrentProcess())
{
r.usedMemory = ((double)proc.PrivateMemorySize64) / 1024 / 1024;
}
r.activeSessions = getServiceData().Count();
string jsCode = "displayData" + "("+
new JavaScriptSerializer().Serialize(r)
+")";
WebOperationContext.Current.OutgoingResponse.ContentType = "application/javascript";
Console.WriteLine(jsCode);
return new MemoryStream(Encoding.UTF8.GetBytes(jsCode));
When used on my devel PC it works fine. I get sth like:
displayData({"_startupTime":"\/Date(1435867525056)\/","serverVersion":"1.0.0.8","startUpTime":"2. 7. 2015 22:05:25","acceptedConnections":0,"upTime":"00.00:00:00","usedMemory":21.265625,"activeSessions":0})
However, after deploy to a remote server I get only following response and I want to get a JSON:
<GetServerInfoResponse xmlns="http://tempuri.org/"><GetServerInfoResult>ZGlzcGxheURhdGEoeyJfc3RhcnR1cFRpbWUiOiJcL0RhdGUoMTQzNTg2ODUwMjc5NClcLyIsInNlcnZlclZlcnNpb24iOiIxLjAuMC44Iiwic3RhcnRVcFRpbWUiOiI3LzIvMjAxNSAxOjIxOjQyIFBNIiwiYWNjZXB0ZWRDb25uZWN0aW9ucyI6MCwidXBUaW1lIjoiMDAuMDA6MDA6MDAiLCJ1c2VkTWVtb3J5IjoyNy40NzY1NjI1LCJhY3RpdmVTZXNzaW9ucyI6MH0p</GetServerInfoResult></GetServerInfoResponse>
Note I call for the request locally directly on the server. But on remote call over network the response is the same. If I put a log output of the string to console I can see the output string is correct.
The config files are identical (except for addresses).
--edit
With the try-fail method I found out the string inside the XML response is the JSON string encoded in Base64.
Can somebody please help me whats wrong?
After some research I did not find any solution. However, after restarting OS (Win Server 2012) and rebuilding it just works like a charm.
We are building a WinRT app which gets data from server which is Web API based & so it gives data in json and/or XML format.
When app user logs in for the first time using his credentials(username,password), the response that comes from server is a success bit & a TOKEN, which should be used in successive URL requests.
I am using httpclient for sending requests
using (HttpClient httpClient1 = new HttpClient())
{
string url = "http://example.com/abc/api/process1/GetLatestdata/10962f61-4865-4e7a-a121-3fdd968824b5?employeeid=6";
//The string 10962f61-4865-4e7a-a121-3fdd968824b5 is the token sent by the server
var response = await httpClient1.GetAsync(new Uri(url));
string content = await response.Content.ReadAsStringAsync();
}
Now the response that i get is with status code 401 "unauthorised".
And the xml i get in response is "Unauthorised User".
Is there anything i need to change in appManifest??
I've checked this, but cant we use httpclient without credentials??
Your Capabilities are enough. You don't even need Internet (Client) because it's included in Internet (Client & Server).
You do not have credentials for WinRT HttpClient, in your linked post they referr to System.Net.Http.HttpClientHandler.
Maybe you can use the HttpBaseProtocolFilter to add the credentials?
using (var httpFilter = new HttpBaseProtocolFilter())
{
using (var httpClient = new HttpClient(httpFilter))
{
httpFilter.ServerCredential...
}
}
I don't know your security mechanism, I'm using a HttpClient and my session-key is in a cookie. But I think your client code looks fine.
I'm using Ember-Data 1.0.0.Beta-9 and Ember 1.7 to consume a REST API via DreamFactory's REST Platform. (http://www.dreamfactory.com).
I've had to extend the RESTAdapter in order to use DF and I've been able to implement GET and POST requests with no problems. I am now trying to implement model.save() (PUT) requests and am having a serious hiccup.
Calling model.save() sends the PUT request with the correct data to my API endpoint and I get a 200 OK response with a JSON response of { "id": "1" } which is what is supposed to happen. However when I try to access the updated record all of the properties are empty except for ID and the record on the server is not updated. I can take the same JSON string passed in the request, paste it into the DreamFactory Swagger API Docs and it works no problem - response is good and the record is updated on the DB.
I've created a JSBin to show all of the code at http://emberjs.jsbin.com/nagoga/1/edit
Unfortunately I can't have a live example as the servers in question are locked down to only accept requests from our company's public IP range.
DreamFactory provides a live demo of the API in question at
https://dsp-sandman1.cloud.dreamfactory.com/swagger/#!/db/replaceRecordsByIds
OK in the end I discovered that you can customize the DreamFactory response by adding a ?fields=* param to the end of the PUT request. I monkey-patched that into my updateRecord method using the following:
updateRecord: function(store, type, record) {
var data = {};
var serializer = store.serializerFor(type.typeKey);
serializer.serializeIntoHash(data, type, record);
var adapter = this;
return new Ember.RSVP.Promise(function(resolve, reject) {
// hack to make DSP send back the full object
adapter.ajax(adapter.buildURL(type.typeKey) + '?fields=*', "PUT", { data: data }).then(function(json){
// if the request is a success we'll return the same data we passed in
resolve(json);
}, function(reason){
reject(reason.responseJSON);
});
});
}
And poof we haz updates!
DreamFactory has support for tacking several params onto the end of the requests to fully customize the response - at some point I will look to implement this correctly but for the time being I can move forward with my project. Yay!
EmberData is interpreting the response from the server as an empty object with an id of "1" an no other properties in it. You need to return the entire new object back from the server with the changes reflected.
Background: I have a WCF Data Service with a Silverlight application that is currently using atom pub xml. I want to use JSON to lessen the size of the payload.
I read that you can JSON from the service webget using the following code:
WebClient wc = new WebClient();
wc.Headers["Accept"] = "application/json";
Can I modify the header for a DataServiceQuery call or a localContext.BeginExecute (for WebGets)?
// WCF Data Services Query Proxy
DataServiceQuery<T> query = filterExpression as DataServiceQuery<T>;
// Execute the ASYNC query against the model
query.BeginExecute(new AsyncCallback((iar) =>
{ ...});
or
// Create new context with the WCF service to force only save this entity
VisiconnEDM localContext = new VisiconnEDM(new Uri(entityServiceURL, UriKind.Absolute));
// execute the query asynchronously
localContext.BeginExecute<T>(urlQuery,(IAsyncResult iar) =>{ ...},null);
Even if you would modify the header for DataServiceRequest the client library of WCF DS doesn't have support for reading JSON responses, so it would not be able to read the response. The currently suggested approach to decrease the payload size is to use GZip.