How to deserialize Json with fix named elements dynamically in vb.net? - json

I’m on the way to implement an interface to the Ameritrade Rest API in a vb.net application (with httpclient).
Amongst other things, I have to query quotes from a ticker list (e.g. AMD,MSFT,AMZN, ....).
The call of the API works without problems, I get a valid Json back, but the Json is not given back in a way, I would expect.
I now search the best way to handle that problem...
This is not the first interface to a Rest API, I have implemented.
Normally, I implement a corresponding data class in vb.net and then use JsonConvert (from Newtonsoft) to deserialize the Json string into my data class.
Example:
Dim oObject As New DataClass
oObject = JsonConvert.DeserializeObject(Of DataClass)(JsonString)
whereby DataClass is the vb.net class that is defined according to the data in the Json string.
Problem:
The ticker symbol-list to query is dynamic and can change from api call to api call.
If I - e.g. - query AMD and MSFT in a call, I get back (cut to only a few fields) the following Json:
{
"AMD": {
"assetType": "EQUITY",
"symbol": "AMD",
"description": "Advanced Micro Devices, Inc. - Common Stock",
"bidPrice": 92.11
},
"MSFT": {
"assetType": "EQUITY",
"symbol": "MSFT",
"description": "Microsoft Corporation - Common Stock",
"bidPrice": 243.1
}
}
To be able to deserialize the Json, I would have to implement the following DataClass:
Public Class DataClass
Public Property AMD As AMD
Public Property MSFT As MSFT
End Class
Public Class AMD
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
Public Class MSFT
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
This would work but is absolutely static and does not make any sense, as I would have to implement a (identical) class for any ticker, I maybe want to query in the feature.
I would expect to get back a dynamic list so that I could implement the class as following:
Public Class DataClass
Public Property TickerDetails As List(Of TickerDetail)
End Class
Public Class TickerDetail
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
This way, I would be able to deserialize in a List of TickerDetails and the go thru the list (no matter, which symbols I queried).
But, I can’t change, what I get back over the API...
Question:
What is the best way to handle this problem?

You should create a class to represent the a generic stock and then use DeserializeObject to deserialize it into a Dictionary(Of String, [classname]) where the Key represents the stock symbol and the value represents the class.
Take a look at this example:
Public Class Stock
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
'...
Dim stocks = JsonConvert.DeserializeObject(Of Dictionary(Of String, Stock))(response)
Example: Live Demo

First thanks for the comments.
I ended up to do it completely different now...
I had further problems with the Ameritrade API:
Some fields are named with leading numbers (52WkHigh and 52WkLow) and
vb.net dev's know, that VB.net don't like properties in classes that
are named with a leading number
So I had to "patch" the received Json data and change the names on the fly to other names ("52WkHigh" to "dble52WkHigh" and "52WkLow" to "dble52WkLow") to be able to deserialize
over the data class, what is not nice
Further, I finally need the data (as fast as possible) in a data table and had "a long way to go":
get data -> deserialize to the data class -> walk thru the data class and overtake the data in the data table.
So.. my new solution (with JObject):
Note: needs:
Imports Newtonsoft.Json.Linq
Code snippets:
Create data table in memory:
Dim dtErgebnis As New DataTable
Dim drTemp As DataRow
With dtErgebnis.Columns
.Add("symbol", System.Type.GetType("System.String"))
.Add("lastPrice", System.Type.GetType("System.Double"))
.Add("lastSize", System.Type.GetType("System.Int32"))
.Add("quoteTime", System.Type.GetType("System.DateTime")) ' Note: is a Long in Json
...
End With
Parse the Json-String and fill the datatable:
get the data over httpclient (in JsonString)...
Dim oJson As JObject = JObject.Parse(JsonString) ' creates children tokens
Dim results As List(Of JToken) = oJson.Children().ToList
For Each item As JProperty In results
item.CreateReader()
drTemp = dtErgebnis.NewRow() ' create a new row to data table in memory
' Fill the fields
drTemp("symbol") = item.Value("symbol")
drTemp("lastPrice") = item.Value("lastPrice")
drTemp("lastSize") = item.Value("lastSize")
drTemp("quoteTime") = GetUTCDateFromTimeStamp(item.Value("quoteTimeInLong")).AddHours(1) ' original Long
...
' Add the new row to the data table
dtErgebnis.Rows.Add(drTemp)
' Save the changes
dtErgebnis.AcceptChanges()
Next
Additional note: The Ameritrade API gives back the time stamps as long (additional hurdle), but I (and I think also you;-) want it as datetime.
Therefore the Long (I think this data type comes from Java/Unix) has to be "translated" to datetime = vb.net function GetUTCDateFromTimeStamp below:
Public Function GetUTCDateFromTimeStamp(TimeStamp As Long) As DateTime
Static startTime As New DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
Return startTime.AddMilliseconds(TimeStamp)
End Function
Additional note:
As I want to have the Swiss time, I add one hour to the UTC time.
So.. this a real good solution for me (exactly for the Ameritrade API).
And.. it's blazing fast... (I get 19 tickers with all fields and show the result (data table) in a data grid).
All together took < 1 Second ("felt" about 500 ms)
Hope this helps somebody...

Related

Get Datatype of Json Entries

I get a Json file from our data provider without any information about the datatype of the data inside. The Json comprises only the data and one entry looks like this (this is survey data and one entry is the data of one person):
{"data":["4482359","12526","2014 Company Y","2","3","1"]}
I deserialized the file and have it within an object now but all entries are strings. I want to import it into a database but each entry should have the datatype that it best fits to it. E.g. 1 is int16, 2014 Company Y is string. Acutally I have only Int16 and String, nothing more. For importing it into the database I acutally need the Sql Server datatype equivalent. Unfortunately I cannot hardcode the datatype into the Table Creation Command since the order of the data is not the same for each query. E.g. First I receive Survey 1 data with the structure shown above. But maybe a few days later I need different data like
{"data":["4482359","2","3","1","12526","2015 Company X"]}
Deserialized the Json Data I have an object (RootObject) with a List(of String) called data. Where the first string is "4482359" the second is "2" the third is "3" and so on.
Public Class RootObject
Public Property data As List(Of String)
End Class
Public Class RootObjectDatatype
Private _rootObject As RootObject
Public Sub New(ByRef rootObject As RootObject)
For Each str As String In rootObject.data
If str > 0 And str < 50000 Then
jsonDatatype.Add("Int")
Else
jsonDatatype.Add("Varchar(" & str.Count & ")")
End If
Next
End Sub
Public Property jsonDatatype As List(Of String)
End Class
But this does not work yet.

How to deserialize theefold nested json data?

Consuming a RestAPI I get a Json file that holds data of surveys. I load the file into the following class:
Public Class GeneralInfo
Public Property id As Integer
Public Property name As String
Public Property data As List(Of FullData)
End Class
Every survey has an id, a name and data.
Public Class FullData
Public Property userID As ULong
Public Property data As List(Of String)
End Class
Every survey holds data for each participant. I use Newtonsoft.Json to get the data into my classes.
JsonConvert.DeserializeObject(Of List(Of GeneralInfo))(jstr)
The data within the data-property is comma separated but the length is different for every survey. So in survey 1 can be 100 datapoints (one for each answer to a question) and in survey 2 can be up to 400 datapoints. A few of these datapoints can include comma as well since there are datapoints that hold open questions.
I just do not know how to store these data properly. I already read into the JsonConvert and could accomplish getting the data into the classes you can see above but here it ends. Could anyone help me out here? I address these to c# as well since I do not care if the solution is c# or vb.net.
Here are some sample data:
[{"id":1,
"name":"2015 - companyname.dpt",
"data":[{
"userId":364,
"data":["367","90","company","","","Anonymous","User","anonymous#provider.org","","100.100.100.100","14242","undefined","0","7891","2","2","3","1","","","","","","","2015","1","","","","","","","","","","","","","","","","3","3","2","1","3","3","3","4","3","3","4","2","3","3","2","3","2","3","4","3","","4","5","3","4","2","4","4","4","4","4","2","4","4","2","2","4","3","4","4","4","4","3","","1","1","1","5","1","1","1","","1","1","1","1","1","3","3","3","3","4","4","1","","","","2","2","3","1","","","","","","","","",""]
}, {"userID": 365, ...}]
}]

Parsing JSON Objects within JSON arrays within JSON Objects in VB.Net

Hello Everyone and thanks for looking at this. I'm relatively new to vb.net and extremely new to parsing json in vb. I am using JSON.Net and I'm looking to gather data from the following JSON.
http://hastebin.com/bagiyetece.apache
I have classes created for each of the "sections". I am unsure of the correct terminology.
Class One:
Public Class StatsWrapper
Public SummonerID as Long
Public PlayerStatSummaries as playerStatSummaryTypeWrapper
End Class
Class Two:
Public Class playerStatSummaryTypeWrapper
Public playerStatSummaryType As String
Public wins As Long
Public modifyDate As Long
Public aggregatedStats As aggregatedStatsWrapper
End Class
Class Three:
http://hastebin.com/qopanafabe.php
My end goal is to be able to get elements like "totalChampionKills" for the playerStatSummaryType of "Cap5x5" and insert them into a datagridview.
I've been able to correctly parse the following JSON simply by using JObject.Parse.
{"UserName":{"id":84737282,"name":"UserName","profileIconId":660,"summonerLevel":30,"revisionDate":1455686689000}}
To get the id object, I would use:
Dim JSONDerulo = JObject.Parse(JSONResponse)
Dim SummonerID = JSONDerulo(LCase(ToolStripTextBox1.Text))("id")
Where the ToolStripTextBox1.Text is the UserName.
When I try to apply the same logic as above to the larger JSON file in a different sub:
Dim JSONDerulo = JObject.Parse(JSONResponse)
Dim PlayerStatSummaries = JSONDerulo("playerStatSummaries")
Dim Jarray As JArray = PlayerStatSummaries
I can do something like:
For Each obj As JObject In Jarray
TextBox2.Text = TextBox2.Text + Environment.NewLine + obj.ToString + Environment.NewLine
Next
Or call Jarray(1) and parse that again but the list of game types is going to be different for each UserName. I do have a master list for every game type though:
AramUnranked5x5, Ascension, Bilgewater, CAP5x5, CoopVsAI, CoopVsAI3x3, CounterPick, FirstBlood1x1, FirstBlood2x2, Hexakill, KingPoro, NightmareBot, OdinUnranked, OneForAll5x5, RankedPremade3x3, RankedPremade5x5, RankedSolo5x5, RankedTeam3x3, RankedTeam5x5, SummonersRift6x6, Unranked, Unranked3x3, URF, URFBots
If I want to (for example) call AramUnranked5x5.TotalChampionKills or something similar would I have to create a class for each type?
Any suggestions or ideas?
Thanks Again
Note: the json posted is considerably shorter and simpler than the gruesome aggregatedStatsWrapper class linked to depicts! Without data, I cant say whether it is right or not.
Since the aggregatedStats is its own type, nothing very interesting will show for it in a DataGridView, just the type name. There are several ways to handle this. One is to hide the property from the DGV, then when they change selected/current rows, find the new one in the list and set player.aggregatedStats as the selected object in a property grid for a master-detail type view. Playerstatsummary:
Public Class Playerstatsummary
Public Property playerStatSummaryType As String
Public Property wins As Integer
Public Property modifyDate As Long
<Browsable(False)>
Public Property aggregatedStats As Aggregatedstats
Public Property losses As Integer
End Class
<Browsable(False)> will result in the TypeName not being shown in a DGV:
Dim jstr = File.ReadAllText("C:\Temp\myjsonfilename")
Dim jobj = JObject.Parse(jstr)
Dim players = JsonConvert.DeserializeObject(Of List(Of Playerstatsummary))(jobj("playerStatSummaries").ToString)
By Parsing it first, you can skip that outer container. jobj("playerStatSummaries").ToString passes the property data to be deserialized into a List.
You can display what you have very easily without having to loop at all:
dgv1.DataSource = players
It wont yet know about Aggregatedstats unless and until you work out that class exactly. Until then, the type name will display. The post mentions being interested in Cap5x5 only. In that case, a PropertyGrid might be a better UI mechanism (after you find that guy in the list). Result:
(This is from before I added <Browsable(False)> to the class). You could show aggregatedStats as detail like this:
Private Sub DataGridView1_SelectionChanged(...etc
If DataGridView1.SelectedRows.Count = 0 Then Return
Dim thisOne = DataGridView1.SelectedRows(0).Cells(0).Value.ToString
Dim player = players.FirstOrDefault(Function(f) f.playerStatSummaryType = thisOne)
If player IsNot Nothing Then
PropertyGrid1.SelectedObject = player.aggregatedStats
End If
End Sub
In case you are wondering, the date is almost certainly a Unix Epoch date, easily converted to .NET.

How do I parse this JSON using VB.net?

So, I'm working on a school project, and I'm trying to figure out the best way to deal with this data file that contains a pretty large amount of JSON objects. I know the basics of VB.net, basic event handling, etc.
I know the basics of designing Structures, and stuff like that, but I need to figure out how to parse and create a list of objects from a 5MB JSON file that contains entries such as the following:
{
"Air Elemental":{
"layout":"normal",
"name":"Air Elemental",
"manaCost":"{3}{U}{U}",
"cmc":5,
"colors":[
"Blue"
],
"type":"Creature — Elemental",
"types":[
"Creature"
],
"subtypes":[
"Elemental"
],
"text":"Flying",
"power":"4",
"toughness":"4",
"imageName":"air elemental"
},
"Ancestral Recall":{
"layout":"normal",
"name":"Ancestral Recall",
"manaCost":"{U}",
"cmc":1,
"colors":[
"Blue"
],
"type":"Instant",
"types":[
"Instant"
],
"text":"Target player draws three cards.",
"imageName":"ancestral recall"
},
"Animate Artifact":{
"layout":"normal",
"name":"Animate Artifact",
"manaCost":"{3}{U}",
"cmc":4,
"colors":[
"Blue"
],
"type":"Enchantment — Aura",
"types":[
"Enchantment"
],
"subtypes":[
"Aura"
],
"text":"Enchant artifact\nAs long as enchanted artifact isn't a creature, it's an artifact creature with power and toughness each equal to its converted mana cost.",
"imageName":"animate artifact"
}
}
If anyone could be of assistance, or just sort of point me in the right direction, I'd really appreciate it. I think the part that's throwing me off the most is that each card name is a key in itself, and all of the card's data is the value associated with the name "key"...
In this case it hardly matters how many there are because they are all structured the same way:
Public Class Spell
Public Property layout As String
Public Property name As String
Public Property manaCost As String
Public Property cmc As Integer
Public Property colors As String()
Public Property type As String
Public Property types As String()
Public Property subtypes As String()
Public Property text As String
Public Property power As String
Public Property toughness As String
Public Property imageName As String
End Class
I have no idea what the data represents, it looks like some fantasy game. When deserialized (I used Newtonsoft), you will end up with a Dictionary of spells with the key being "Air Elemental" and "Animate Artifact" etc. It takes but one line of code:
Dim jstr As String = from whereever
Dim mySpells = JsonConvert.DeserializeObject(Of Dictionary(Of String, Spell))(jstr)
Using the NET JavaScriptSerializer is about the same:
Dim jss As New JavaScriptSerializer
Dim myspells = jss.DeserializeObject(jstr)
You can use http://jsonutils.com/, or even Paste Special in VS, to help parse what the structures (classes, really) need to look like. However, it pays to use your brain and look at them. A robot will create 3 classes for that JSON and another class container to hold them. If there were 100 of them it would be needlessly long.
Since each item is identical and the JSON one class can be used for each. Since the JSON is formatted to accommodate, a standard NET dictionary works fine.
There is a wonderful feature in VS 2013 called "Paste as JSON" under the Edit menu, so copy your JSON string and choose this feature.
Be aware that there is a small bug that I reported to MS in the way that it declares arrays. The text it gives you will be
Public Property x() as DataType
however it needs to be changed to
Public Property x as DataType()
in order to be correctly declared as an array.

ArgumentOutOfRange exception when deserializing to a DataSet with Json.Net

I have the following JSON string in a variable called strJSON.
{
"results":[
{
"templateName":"HUD Section 8",
"userID":"2",
"mobileObjectId":"4582",
"source":"M",
"inspectionType":"A",
"notes":"Window in bedroom needs repair.",
"agencyID":"",
"requestDate":"2014-05-09 00:00:00",
"agencyName":"",
"inspectionTimeBegun":"2014-05-09 14:00:17",
"inspectionDate":"2014-05-09 14:30:00",
"inspectionID":135,
"inspectionTimeComplete":"2014-05-09 14:29:25",
"summaryDecision":"F",
"createdAt":"2014-05-09T18:29:35.050Z",
"updatedAt":"2014-05-09T18:29:35.050Z",
"objectId":"1FgtD6WT8Y",
"ACL":{
"*":{
"read":true
},
"cryZoU5gXJ":{
"write":true,
"read":true
}
}
}
]
}
When I call the following line of code...
ds = Newtonsoft.Json.JsonConvert.DeserializeObject(Of DataSet)(strJSON)
I get an exception with the message stating "Specified argument was out of the range of valid values"
The JSON string is created with the following REST API call to Parse.com.
strJSON = http.QuickGetStr(strURL)
I am using this elsewhere with success albeit with simpler Parse classes but I have gone through this JSON string carefully and can't see anything wrong.
Any ideas on what might be causing this error?
In order for Json.Net to deserialize into a DataSet, the JSON must be in a specific format, as described in this answer. Your JSON is close, but the problem is the ACL object. The DataTableConverter that Json.Net 5.0 uses expects all of the columns in the table to be simple data types or it will throw an ArgumentOutOfRangeException (source). Json.Net 6.0 supports nested data tables and arrays in addition to simple types, but your ACL data still does not meet the required format that would allow it to be deserialized correctly to a DataSet. You have a few different options for dealing with this:
Change the JSON
If you control the format of the JSON (i.e. it is not from a third party) you can change it such that Json.Net 6.0 will be able to deserialize it to a DataSet. Here is what it would need to look like for that to work:
{
"results": [
{
"templateName": "HUD Section 8",
"userID": "2",
"mobileObjectId": "4582",
"source": "M",
"inspectionType": "A",
"notes": "Window in bedroom needs repair.",
"agencyID": "",
"requestDate": "2014-05-09 00:00:00",
"agencyName": "",
"inspectionTimeBegun": "2014-05-09 14:00:17",
"inspectionDate": "2014-05-09 14:30:00",
"inspectionID": 135,
"inspectionTimeComplete": "2014-05-09 14:29:25",
"summaryDecision": "F",
"createdAt": "2014-05-09T18:29:35.050Z",
"updatedAt": "2014-05-09T18:29:35.050Z",
"objectId": "1FgtD6WT8Y",
"ACL": [
{
"user": "*",
"read": true,
"write": false
},
{
"user": "cryZoU5gXJ",
"read": true,
"write": true
}
]
}
]
}
With this format, the ACL column of the results table will contain a nested DataTable with the individual ACL rows, each row having three columns, user, read and write.
Deserialize to strongly-typed classes
Instead of deserializing into a DataSet, you could deserialize into a set of strongly-typed classes. The advantage to this approach is that everything is in an easily usable form. The disadvantage is that you need to know what is in the JSON before you can create the classes.
You can use third-party tools like json2csharp.com to help generate the classes from a sample of the JSON, as was suggested in another answer (now deleted), but note that this is not foolproof (and it doesn't do VB). Sometimes you will need to intervene and edit the classes manually. For example, if generate classes from the JSON in your question, you'll notice that it creates a fixed class for each ACL instance. This will not work unless your set of ACLs always has exactly two items, one called Everyone and the other CryZoU5gXJ. I think it is much more likely that the set of ACLs will be variable, so it makes sense to use a Dictionary for these. Here are the classes I would propose:
Class RootObject
Public Property results As List(Of Result)
End Class
Class Result
Public Property templateName As String
Public Property userID As String
Public Property mobileObjectId As String
Public Property source As String
Public Property inspectionType As String
Public Property notes As String
Public Property agencyID As String
Public Property requestDate As String
Public Property agencyName As String
Public Property inspectionTimeBegun As String
Public Property inspectionDate As String
Public Property inspectionID As Integer
Public Property inspectionTimeComplete As String
Public Property summaryDecision As String
Public Property createdAt As String
Public Property updatedAt As String
Public Property objectId As String
Public Property ACL As Dictionary(Of String, ACL)
End Class
Class ACL
Public Property read As Boolean
Public Property write As Boolean
End Class
With this class structure in place, you can deserialize like this:
Dim root As RootObject = JsonConvert.DeserializeObject(Of RootObject)(strJSON)
For the ACLs, the key for each dictionary entry will be the user ID (or * as you have in your example). If you don't actually care about the ACLs, you can simply omit the ACL property from the Result class. By default Json.Net will skip properties that exist in the JSON but do not exist in the class.
Use the LINQ-to-JSON API to parse the JSON
With Json.Net there is always more than one way to skin the cat. Json.Net's LINQ-to-JSON API really shines when the JSON you are parsing is highly variable and/or you don't want to create classes for receiving the data. You can deserialize any valid JSON to a hierarchy of JToken objects and then pick them apart however you need. For example, if you just needed a few select pieces of information from each result, you could do this:
Dim token As JToken = JToken.Parse(json)
For Each result As JObject In token("results").Children(Of JObject)()
Console.WriteLine("userID: " + result("userID").ToString())
Console.WriteLine("templateName: " + result("templateName").ToString())
Console.WriteLine("inspectionID: " + result("inspectionID").ToString())
Console.WriteLine("inspectionType: " + result("inspectionType").ToString())
Console.WriteLine("inspectionDate: " + result("inspectionDate").ToString())
Console.WriteLine("summaryDecision: " + result("summaryDecision").ToString())
Console.WriteLine("notes: " + result("notes").ToString())
Next
You could use this same approach to manually build a DataSet from the JSON. Here is a generic function that will deserialize JSON into a DataSet but ignore any complex objects (e.g. the ACLs) instead of throwing an exception:
Function DeserializeToDataSet(json As String) As DataSet
Dim root As JObject = JObject.Parse(json)
Dim ds As DataSet = New DataSet()
For Each prop As JProperty In root.Properties
If prop.Value.Type = JTokenType.Array Then
Dim dt As DataTable = ds.Tables.Add(prop.Name)
For Each row As JObject In prop.Value.Children(Of JObject)()
Dim dr As DataRow = dt.NewRow
For Each col As JProperty In row.Properties
Dim colType As Type = GetColumnType(col.Value.Type)
If Not colType Is Nothing Then
Dim dc As DataColumn = dt.Columns(col.Name)
If dc Is Nothing Then
dc = dt.Columns.Add(col.Name, colType)
End If
dr(col.Name) = col.Value.ToObject(colType)
End If
Next
dt.Rows.Add(dr)
Next
End If
Next
Return ds
End Function
Function GetColumnType(tokenType As JTokenType) As Type
If tokenType = JTokenType.String Then Return GetType(String)
If tokenType = JTokenType.Integer Then Return GetType(Integer)
If tokenType = JTokenType.Date Then Return GetType(DateTime)
If tokenType = JTokenType.Boolean Then Return GetType(Boolean)
If tokenType = JTokenType.Float Then Return GetType(Double)
Return Nothing
End Function
Of course if you need the ACLs, you'll need to customize this method to get that data into a form that is consumable by your code. I'll leave that part to you.
Json.Net will only parse directly into a DataSet if it conforms to a certain standard. See this answer for the layout it needs.
However, you could deserialize to an XML document and use the DataSet object's ReadXml method load it for you. See this question for details on how to do this.
(HT to Brian Rogers for dataset structure details)