Related
I’m on the way to implement an interface to the Ameritrade Rest API in a vb.net application (with httpclient).
Amongst other things, I have to query quotes from a ticker list (e.g. AMD,MSFT,AMZN, ....).
The call of the API works without problems, I get a valid Json back, but the Json is not given back in a way, I would expect.
I now search the best way to handle that problem...
This is not the first interface to a Rest API, I have implemented.
Normally, I implement a corresponding data class in vb.net and then use JsonConvert (from Newtonsoft) to deserialize the Json string into my data class.
Example:
Dim oObject As New DataClass
oObject = JsonConvert.DeserializeObject(Of DataClass)(JsonString)
whereby DataClass is the vb.net class that is defined according to the data in the Json string.
Problem:
The ticker symbol-list to query is dynamic and can change from api call to api call.
If I - e.g. - query AMD and MSFT in a call, I get back (cut to only a few fields) the following Json:
{
"AMD": {
"assetType": "EQUITY",
"symbol": "AMD",
"description": "Advanced Micro Devices, Inc. - Common Stock",
"bidPrice": 92.11
},
"MSFT": {
"assetType": "EQUITY",
"symbol": "MSFT",
"description": "Microsoft Corporation - Common Stock",
"bidPrice": 243.1
}
}
To be able to deserialize the Json, I would have to implement the following DataClass:
Public Class DataClass
Public Property AMD As AMD
Public Property MSFT As MSFT
End Class
Public Class AMD
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
Public Class MSFT
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
This would work but is absolutely static and does not make any sense, as I would have to implement a (identical) class for any ticker, I maybe want to query in the feature.
I would expect to get back a dynamic list so that I could implement the class as following:
Public Class DataClass
Public Property TickerDetails As List(Of TickerDetail)
End Class
Public Class TickerDetail
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
This way, I would be able to deserialize in a List of TickerDetails and the go thru the list (no matter, which symbols I queried).
But, I can’t change, what I get back over the API...
Question:
What is the best way to handle this problem?
You should create a class to represent the a generic stock and then use DeserializeObject to deserialize it into a Dictionary(Of String, [classname]) where the Key represents the stock symbol and the value represents the class.
Take a look at this example:
Public Class Stock
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
'...
Dim stocks = JsonConvert.DeserializeObject(Of Dictionary(Of String, Stock))(response)
Example: Live Demo
First thanks for the comments.
I ended up to do it completely different now...
I had further problems with the Ameritrade API:
Some fields are named with leading numbers (52WkHigh and 52WkLow) and
vb.net dev's know, that VB.net don't like properties in classes that
are named with a leading number
So I had to "patch" the received Json data and change the names on the fly to other names ("52WkHigh" to "dble52WkHigh" and "52WkLow" to "dble52WkLow") to be able to deserialize
over the data class, what is not nice
Further, I finally need the data (as fast as possible) in a data table and had "a long way to go":
get data -> deserialize to the data class -> walk thru the data class and overtake the data in the data table.
So.. my new solution (with JObject):
Note: needs:
Imports Newtonsoft.Json.Linq
Code snippets:
Create data table in memory:
Dim dtErgebnis As New DataTable
Dim drTemp As DataRow
With dtErgebnis.Columns
.Add("symbol", System.Type.GetType("System.String"))
.Add("lastPrice", System.Type.GetType("System.Double"))
.Add("lastSize", System.Type.GetType("System.Int32"))
.Add("quoteTime", System.Type.GetType("System.DateTime")) ' Note: is a Long in Json
...
End With
Parse the Json-String and fill the datatable:
get the data over httpclient (in JsonString)...
Dim oJson As JObject = JObject.Parse(JsonString) ' creates children tokens
Dim results As List(Of JToken) = oJson.Children().ToList
For Each item As JProperty In results
item.CreateReader()
drTemp = dtErgebnis.NewRow() ' create a new row to data table in memory
' Fill the fields
drTemp("symbol") = item.Value("symbol")
drTemp("lastPrice") = item.Value("lastPrice")
drTemp("lastSize") = item.Value("lastSize")
drTemp("quoteTime") = GetUTCDateFromTimeStamp(item.Value("quoteTimeInLong")).AddHours(1) ' original Long
...
' Add the new row to the data table
dtErgebnis.Rows.Add(drTemp)
' Save the changes
dtErgebnis.AcceptChanges()
Next
Additional note: The Ameritrade API gives back the time stamps as long (additional hurdle), but I (and I think also you;-) want it as datetime.
Therefore the Long (I think this data type comes from Java/Unix) has to be "translated" to datetime = vb.net function GetUTCDateFromTimeStamp below:
Public Function GetUTCDateFromTimeStamp(TimeStamp As Long) As DateTime
Static startTime As New DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
Return startTime.AddMilliseconds(TimeStamp)
End Function
Additional note:
As I want to have the Swiss time, I add one hour to the UTC time.
So.. this a real good solution for me (exactly for the Ameritrade API).
And.. it's blazing fast... (I get 19 tickers with all fields and show the result (data table) in a data grid).
All together took < 1 Second ("felt" about 500 ms)
Hope this helps somebody...
I have a dataframe. I need to call a Rest API for each record.
Lets say the Dataframe looks like:
|----|-------------|-----|---------|
|UUID|PID |DEVID|FIRSTNAME|
|----|-------------|-----|---------|
|1111|1234567891011|ABC11|JOHN |
|2222|9876543256827|ABC22|HARRY |
|----|-------------|-----|---------|
The JSON request string for first row should look like(Note: the json is created on 2 columns, not all), as the Rest API to be called requires the input in this format:
{"applicationInfo": {"appId": "ec78fef4-92b9-3b1b-a68d-c45376b6977a"}, "requestData": [{"secureData": "JOHN", "secureDataType": "FIRSTNAME", "index": 1 }, {"secureData": "1234567891011", "secureDataType": "PID", "index": 2 } ] }
The value of index key has to be generted on the fly, using an incremental counter for each row.
Then, i need to call the Rest API sending the above JSON as a string param.
The response from the API after encryption will look like:
{"responseData":[{"resultCode":"00","secureData":"63ygdydshbhgvdyw3et7edgu","secureDataType":"FIRSTNAME","index":1},{"resultCode":"00","secureData":"HKJJBJHVHG66456456FXXFFCGF","secureDataType":"PID","index":2}],"responseCode":"00","responseMessage":"SUCCESS","resultCounts":{"totalCount":2,"successCount":2,"failedCount":0}}
Then I need to read the above response and create a dataframe which should look like:
|----|--------------------------|-----|------------------------|
|UUID|PID |DEVID|FIRSTNAME |
|----|--------------------------|-----|------------------------|
|1111|HKJJBJHVHG66456456FXXFFCGF|ABC11|63ygdydshbhgvdyw3et7edgu|
|----|--------------------------|-----|------------------------|
If i convert the initial input dataframe toJSON().collectAsList(), then it looks like:
[{"UUID":"1111","PID":"1234567891011","DEVID":"ABC11","FIRSTNAME":"JOHN"}, {"UUID":"2222","PID":"9876543256827","DEVID":"ABC22","FIRSTNAME":"HARRY"}]
But this doesnt work as the Rest API requires its input in a certain format, mentioned above.
Please help.
For the above, I assume that the data set has been partitioned across the number of Spark workers and it is a generic data set of Row (data frame), then the below mechanism can be employed.
Define a class with the required attributes as a data container
Take the data set content as a List (takeAsList method if data set, refer)
Create and populate the objects of your data container (and store in such a way to identify them later, you shall have to repopulate them with decrypted data)
Serialize the list into a JSON array with Jackson (refer)
Step 4 & 5 can be combined with Jackson custom serializer refer example
Make the REST call and repopulate the data container objects (after deserializing the response with Jackson)
Create a data frame (an example)
Process the data frame (dataset of rows)
NOTE: The JSON structure you have provided seems not to be correct, JSON array is [{},{},{}]
In your case, given the format of the request JSON, direct conversion of rows will not work, as mentioned in point 1, make a set of model classes, you could consider the below model classes.
package org.test.json;
import java.util.List;
public class RequestModel {
protected ApplicationInfo applicationInfo;
protected List<RequestData> requestData;
public ApplicationInfo getApplicationInfo() {return applicationInfo;}
public void setApplicationInfo(ApplicationInfo applicationInfo) {this.applicationInfo = applicationInfo;}
public List<RequestData> getRequestData() {return requestData;}
public void setRequestData(List<RequestData> requestData) {this.requestData = requestData;}
}//class closing
package org.test.json;
public class ApplicationInfo {
protected String appId;
public String getAppId() {return appId;}
public void setAppId(String appId) {this.appId = appId;}
}//class closing
package org.test.json;
public class RequestData {
protected String secureData;
protected String secureDataType;
protected int index;
public String getSecureData() {return secureData;}
public void setSecureData(String secureData) {this.secureData = secureData;}
public String getSecureDataType() {return secureDataType;}
public void setSecureDataType(String secureDataType) {this.secureDataType = secureDataType;}
public int getIndex() {return index;}
public void setIndex(int index) {this.index = index;}
}//class closing
Process the list as obtained from the data frame and populate the model classes and then convert with Jackson to get the request JSON.
The below should do what you are looking for, don't directly run this, the data set is null
//Do not run this, will generate NullPointer, for example only
Dataset<Row> ds=null;
List<Row> rows=ds.collectAsList();
RequestModel request=new RequestModel();
//Set application id
ApplicationInfo appInfo=new ApplicationInfo();
appInfo.setAppId("some id");
request.setApplicationInfo(appInfo);
List<RequestData> reqData=new ArrayList<>();
for(int i=0;i<rows.size();i++) {
//Incrementally generated for each row
int index=i;
Row r=rows.get(i);
int rowLength=r.size();
for(int j=0;j<rowLength;j++) {
RequestData dataElement=new RequestData();
dataElement.setIndex(index);
switch(j) {
case 1:{dataElement.setSecureData(r.getString(j));dataElement.setSecureDataType("PID");break;}
case 3:{dataElement.setSecureDataType(r.getString(j));dataElement.setSecureDataType("FIRSTNAME");break;}
default:{break;}
}//switch closing
reqData.add(dataElement);
}//for closing
}//for closing
I have a very simple application that I am using to learn move about MVC. In that application I search some data and return a JSON string that I want to use as the model for the view. I just can't seem to figure out how to get the view to consume the JSON as a set of data for it to show on screen.
The code I have so far:
Function Find(term As String) As ActionResult
Dim model As String = SearchData(term)
Return View(model)
End Function
SearchData returns a JSON string that can have one or many objects in it.
How do I now take the JSON returned from SearchData and use it in a view? In fact, when I attempt to add a view it wants to know what model to use. How do i also specific that?
Create a strongly typed object to store you data when parsed
Public Class Data
Public Property ID As Integer
Public Property Term As String
Public Property SomeProperty As String
Public Property SomeOtherProperty As String
End Class
Using a library like JSON.Net, parse the JSON returned from the search.
This assumes a collection of Data is returned from the search.
Imports Newtonsoft.Json;
Function Find(term As String) As ActionResult
Dim json As String = SearchData(term)
Dim model As List(Of Data) = JsonConvert.DeserializeObject(Of List(Of Data))(json)
Return View(model)
End Function
Let the view know to expect the strongly typed model.
#ModelType List(Of Data)
#Code
ViewData("Title") = "Find"
End Code
<h2>MyView</h2>
<!-- rest of view where model can be accessed -->
I've read other answers, but I still seem lost. I am trying to bring JSON into my objects. I call a JSON string through an API, and I want to convert it into the objects after the RequestManager. Am I missing methods or something? Also, the returnChampions2 function is wrong. I posted part of the JSON string after the methods, it's really long.
Public Class RequestManager
Public Function returnChampions2(ByVal strRegion As String) As LeagueChampionMaster
Dim strRequest As String = "https://global.api.pvp.net/api/lol/static-data/" + strRegion + "/v1.2/champion?api_key=" + _APIKey
Return JsonConvert.DeserializeObject(Of LeagueChampionMaster)(returnJSONRequest(strRequest))
End Function
Public Class LeagueChampionMaster
Private _Type As String
Private _Version As String
Private _Data As LeagueChampion()
End Class
Public Class LeagueChampion
Private _ID As Integer
Private _Key As String
Private _Name As String
Private _Title As String
End Class
The json:
{"type":"champion","version":"6.24.1","data":{"Jax":{"id":24,"key":"Jax","name":"Jax","title":"Grandmaster
at Arms"},"Sona":{"id":37,"key":"Sona","name":"Sona","title":"Maven of
the
Strings"},"Tristana":{"id":18,"key":"Tristana","name":"Tristana","title":"the
Yordle
Gunner"},"Varus":{"id":110,"key":"Varus","name":"Varus","title":"the
Arrow of
Retribution"},"Fiora":{"id":114,"key":"Fiora","name":"Fiora","title":"the
Grand
Duelist"},"Singed":{"id":27,"key":"Singed","name":"Singed","title":"the
Mad Chemist"},"TahmKench":{"id":223,"key":"TahmKench","name":"Tahm
Kench","title":"the River
King"},"Leblanc":{"id":7,"key":"Leblanc","name":"LeBlanc","title":"the
Deceiver"},"Thresh":{"id":412,"key":"Thresh","name":"Thresh","title":"the
Chain
Warden"},"Karma":{"id":43,"key":"Karma","name":"Karma","title":"the
Enlightened
One"},"Jhin":{"id":202,"key":"Jhin","name":"Jhin","title":"the
Virtuoso"},"Rumble":{"id":68,"key":"Rumble","name":"Rumble","title":"the
Mechanized
Menace"},"Udyr":{"id":77,"key":"Udyr","name":"Udyr","title":"the
Spirit Walker"},"LeeSin":{"id":64,"key":"LeeSin","name":"Lee
Sin","title":"the Blind
Monk"},"Yorick":{"id":83,"key":"Yorick","name":"Yorick","title":"Shepherd
of
Souls"},"Kassadin":{"id":38,"key":"Kassadin","name":"Kassadin","title":"the
Void
Walker"},"Sivir":{"id":15,"key":"Sivir","name":"Sivir","title":"the
Battle
Mistress"},"MissFortune":{"id":21,"key":"MissFortune","name":"Miss
Fortune","title":"the Bounty
Hunter"},"Draven":{"id":119,"key":"Draven","name":"Draven","title":"the
Glorious
Executioner"},"Yasuo":{"id":157,"key":"Yasuo","name":"Yasuo","title":"the
Unforgiven"},"Kayle":{"id":10,"key":"Kayle","name":"Kayle","title":"The
Judicator"},"Shaco":{"id":35,"key":"Shaco","name":"Shaco","title":"the
Demon
Jester"},"Renekton":{"id":58,"key":"Renekton","name":"Renekton","title":"the
Butcher of the
Sands"},"Hecarim":{"id":120,"key":"Hecarim","name":"Hecarim","title":"the
Shadow of
War"},"Fizz":{"id":105,"key":"Fizz","name":"Fizz","title":"the Tidal
Trickster"}}}
You have several issues. You should know that if you copy the json to the clipboard, Edit -> Paste Special -> Paste Json as Classes Visual Studio will create the classes for you to give you a decent starting point. In this case, the tool is a little dense and will create umpteen identical classes for "Fizz", "Shaco" etc. You have already normalized that.
However, with all the properties Private you wont be able to access the data. Then, the property names are wrong. The json key of id or key will not map/deserialize to _id or _key because they do not match. Lastly, the data collection should be a Dictionary. The key for each player/champion/item will be used as the Dictionary key:
Public Class LeagueContainer
Public Property type As String
Public Property version As String
Public Property data As Dictionary(Of String, DataItem)
End Class
Public Class DataItem
Public Property id As Integer
Public Property key As String
Public Property name As String
Public Property title As String
End Class
usage:
Dim jstr = ... from where ever ...
Dim myData = JsonConvert.DeserializeObject(Of LeagueContainer)(jstr)
' print the keys
For Each kvp In myData.data
Console.WriteLine(kvp.Key)
Next
' what is Yorick's title?
Console.WriteLine("Yorick is '{0}'", myData.data("Yorick").title)
(Partial) Output:
Jax
Sona
Tristana
Varus
Fiora
...
Yorick is 'Shepherd of Souls'
I have the following JSON string in a variable called strJSON.
{
"results":[
{
"templateName":"HUD Section 8",
"userID":"2",
"mobileObjectId":"4582",
"source":"M",
"inspectionType":"A",
"notes":"Window in bedroom needs repair.",
"agencyID":"",
"requestDate":"2014-05-09 00:00:00",
"agencyName":"",
"inspectionTimeBegun":"2014-05-09 14:00:17",
"inspectionDate":"2014-05-09 14:30:00",
"inspectionID":135,
"inspectionTimeComplete":"2014-05-09 14:29:25",
"summaryDecision":"F",
"createdAt":"2014-05-09T18:29:35.050Z",
"updatedAt":"2014-05-09T18:29:35.050Z",
"objectId":"1FgtD6WT8Y",
"ACL":{
"*":{
"read":true
},
"cryZoU5gXJ":{
"write":true,
"read":true
}
}
}
]
}
When I call the following line of code...
ds = Newtonsoft.Json.JsonConvert.DeserializeObject(Of DataSet)(strJSON)
I get an exception with the message stating "Specified argument was out of the range of valid values"
The JSON string is created with the following REST API call to Parse.com.
strJSON = http.QuickGetStr(strURL)
I am using this elsewhere with success albeit with simpler Parse classes but I have gone through this JSON string carefully and can't see anything wrong.
Any ideas on what might be causing this error?
In order for Json.Net to deserialize into a DataSet, the JSON must be in a specific format, as described in this answer. Your JSON is close, but the problem is the ACL object. The DataTableConverter that Json.Net 5.0 uses expects all of the columns in the table to be simple data types or it will throw an ArgumentOutOfRangeException (source). Json.Net 6.0 supports nested data tables and arrays in addition to simple types, but your ACL data still does not meet the required format that would allow it to be deserialized correctly to a DataSet. You have a few different options for dealing with this:
Change the JSON
If you control the format of the JSON (i.e. it is not from a third party) you can change it such that Json.Net 6.0 will be able to deserialize it to a DataSet. Here is what it would need to look like for that to work:
{
"results": [
{
"templateName": "HUD Section 8",
"userID": "2",
"mobileObjectId": "4582",
"source": "M",
"inspectionType": "A",
"notes": "Window in bedroom needs repair.",
"agencyID": "",
"requestDate": "2014-05-09 00:00:00",
"agencyName": "",
"inspectionTimeBegun": "2014-05-09 14:00:17",
"inspectionDate": "2014-05-09 14:30:00",
"inspectionID": 135,
"inspectionTimeComplete": "2014-05-09 14:29:25",
"summaryDecision": "F",
"createdAt": "2014-05-09T18:29:35.050Z",
"updatedAt": "2014-05-09T18:29:35.050Z",
"objectId": "1FgtD6WT8Y",
"ACL": [
{
"user": "*",
"read": true,
"write": false
},
{
"user": "cryZoU5gXJ",
"read": true,
"write": true
}
]
}
]
}
With this format, the ACL column of the results table will contain a nested DataTable with the individual ACL rows, each row having three columns, user, read and write.
Deserialize to strongly-typed classes
Instead of deserializing into a DataSet, you could deserialize into a set of strongly-typed classes. The advantage to this approach is that everything is in an easily usable form. The disadvantage is that you need to know what is in the JSON before you can create the classes.
You can use third-party tools like json2csharp.com to help generate the classes from a sample of the JSON, as was suggested in another answer (now deleted), but note that this is not foolproof (and it doesn't do VB). Sometimes you will need to intervene and edit the classes manually. For example, if generate classes from the JSON in your question, you'll notice that it creates a fixed class for each ACL instance. This will not work unless your set of ACLs always has exactly two items, one called Everyone and the other CryZoU5gXJ. I think it is much more likely that the set of ACLs will be variable, so it makes sense to use a Dictionary for these. Here are the classes I would propose:
Class RootObject
Public Property results As List(Of Result)
End Class
Class Result
Public Property templateName As String
Public Property userID As String
Public Property mobileObjectId As String
Public Property source As String
Public Property inspectionType As String
Public Property notes As String
Public Property agencyID As String
Public Property requestDate As String
Public Property agencyName As String
Public Property inspectionTimeBegun As String
Public Property inspectionDate As String
Public Property inspectionID As Integer
Public Property inspectionTimeComplete As String
Public Property summaryDecision As String
Public Property createdAt As String
Public Property updatedAt As String
Public Property objectId As String
Public Property ACL As Dictionary(Of String, ACL)
End Class
Class ACL
Public Property read As Boolean
Public Property write As Boolean
End Class
With this class structure in place, you can deserialize like this:
Dim root As RootObject = JsonConvert.DeserializeObject(Of RootObject)(strJSON)
For the ACLs, the key for each dictionary entry will be the user ID (or * as you have in your example). If you don't actually care about the ACLs, you can simply omit the ACL property from the Result class. By default Json.Net will skip properties that exist in the JSON but do not exist in the class.
Use the LINQ-to-JSON API to parse the JSON
With Json.Net there is always more than one way to skin the cat. Json.Net's LINQ-to-JSON API really shines when the JSON you are parsing is highly variable and/or you don't want to create classes for receiving the data. You can deserialize any valid JSON to a hierarchy of JToken objects and then pick them apart however you need. For example, if you just needed a few select pieces of information from each result, you could do this:
Dim token As JToken = JToken.Parse(json)
For Each result As JObject In token("results").Children(Of JObject)()
Console.WriteLine("userID: " + result("userID").ToString())
Console.WriteLine("templateName: " + result("templateName").ToString())
Console.WriteLine("inspectionID: " + result("inspectionID").ToString())
Console.WriteLine("inspectionType: " + result("inspectionType").ToString())
Console.WriteLine("inspectionDate: " + result("inspectionDate").ToString())
Console.WriteLine("summaryDecision: " + result("summaryDecision").ToString())
Console.WriteLine("notes: " + result("notes").ToString())
Next
You could use this same approach to manually build a DataSet from the JSON. Here is a generic function that will deserialize JSON into a DataSet but ignore any complex objects (e.g. the ACLs) instead of throwing an exception:
Function DeserializeToDataSet(json As String) As DataSet
Dim root As JObject = JObject.Parse(json)
Dim ds As DataSet = New DataSet()
For Each prop As JProperty In root.Properties
If prop.Value.Type = JTokenType.Array Then
Dim dt As DataTable = ds.Tables.Add(prop.Name)
For Each row As JObject In prop.Value.Children(Of JObject)()
Dim dr As DataRow = dt.NewRow
For Each col As JProperty In row.Properties
Dim colType As Type = GetColumnType(col.Value.Type)
If Not colType Is Nothing Then
Dim dc As DataColumn = dt.Columns(col.Name)
If dc Is Nothing Then
dc = dt.Columns.Add(col.Name, colType)
End If
dr(col.Name) = col.Value.ToObject(colType)
End If
Next
dt.Rows.Add(dr)
Next
End If
Next
Return ds
End Function
Function GetColumnType(tokenType As JTokenType) As Type
If tokenType = JTokenType.String Then Return GetType(String)
If tokenType = JTokenType.Integer Then Return GetType(Integer)
If tokenType = JTokenType.Date Then Return GetType(DateTime)
If tokenType = JTokenType.Boolean Then Return GetType(Boolean)
If tokenType = JTokenType.Float Then Return GetType(Double)
Return Nothing
End Function
Of course if you need the ACLs, you'll need to customize this method to get that data into a form that is consumable by your code. I'll leave that part to you.
Json.Net will only parse directly into a DataSet if it conforms to a certain standard. See this answer for the layout it needs.
However, you could deserialize to an XML document and use the DataSet object's ReadXml method load it for you. See this question for details on how to do this.
(HT to Brian Rogers for dataset structure details)