Get Datatype of Json Entries - json

I get a Json file from our data provider without any information about the datatype of the data inside. The Json comprises only the data and one entry looks like this (this is survey data and one entry is the data of one person):
{"data":["4482359","12526","2014 Company Y","2","3","1"]}
I deserialized the file and have it within an object now but all entries are strings. I want to import it into a database but each entry should have the datatype that it best fits to it. E.g. 1 is int16, 2014 Company Y is string. Acutally I have only Int16 and String, nothing more. For importing it into the database I acutally need the Sql Server datatype equivalent. Unfortunately I cannot hardcode the datatype into the Table Creation Command since the order of the data is not the same for each query. E.g. First I receive Survey 1 data with the structure shown above. But maybe a few days later I need different data like
{"data":["4482359","2","3","1","12526","2015 Company X"]}
Deserialized the Json Data I have an object (RootObject) with a List(of String) called data. Where the first string is "4482359" the second is "2" the third is "3" and so on.
Public Class RootObject
Public Property data As List(Of String)
End Class
Public Class RootObjectDatatype
Private _rootObject As RootObject
Public Sub New(ByRef rootObject As RootObject)
For Each str As String In rootObject.data
If str > 0 And str < 50000 Then
jsonDatatype.Add("Int")
Else
jsonDatatype.Add("Varchar(" & str.Count & ")")
End If
Next
End Sub
Public Property jsonDatatype As List(Of String)
End Class
But this does not work yet.

Related

How to deserialize Json with fix named elements dynamically in vb.net?

I’m on the way to implement an interface to the Ameritrade Rest API in a vb.net application (with httpclient).
Amongst other things, I have to query quotes from a ticker list (e.g. AMD,MSFT,AMZN, ....).
The call of the API works without problems, I get a valid Json back, but the Json is not given back in a way, I would expect.
I now search the best way to handle that problem...
This is not the first interface to a Rest API, I have implemented.
Normally, I implement a corresponding data class in vb.net and then use JsonConvert (from Newtonsoft) to deserialize the Json string into my data class.
Example:
Dim oObject As New DataClass
oObject = JsonConvert.DeserializeObject(Of DataClass)(JsonString)
whereby DataClass is the vb.net class that is defined according to the data in the Json string.
Problem:
The ticker symbol-list to query is dynamic and can change from api call to api call.
If I - e.g. - query AMD and MSFT in a call, I get back (cut to only a few fields) the following Json:
{
"AMD": {
"assetType": "EQUITY",
"symbol": "AMD",
"description": "Advanced Micro Devices, Inc. - Common Stock",
"bidPrice": 92.11
},
"MSFT": {
"assetType": "EQUITY",
"symbol": "MSFT",
"description": "Microsoft Corporation - Common Stock",
"bidPrice": 243.1
}
}
To be able to deserialize the Json, I would have to implement the following DataClass:
Public Class DataClass
Public Property AMD As AMD
Public Property MSFT As MSFT
End Class
Public Class AMD
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
Public Class MSFT
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
This would work but is absolutely static and does not make any sense, as I would have to implement a (identical) class for any ticker, I maybe want to query in the feature.
I would expect to get back a dynamic list so that I could implement the class as following:
Public Class DataClass
Public Property TickerDetails As List(Of TickerDetail)
End Class
Public Class TickerDetail
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
This way, I would be able to deserialize in a List of TickerDetails and the go thru the list (no matter, which symbols I queried).
But, I can’t change, what I get back over the API...
Question:
What is the best way to handle this problem?
You should create a class to represent the a generic stock and then use DeserializeObject to deserialize it into a Dictionary(Of String, [classname]) where the Key represents the stock symbol and the value represents the class.
Take a look at this example:
Public Class Stock
Public Property assetType As String
Public Property symbol As String
Public Property description As String
Public Property bidPrice As Double
End Class
'...
Dim stocks = JsonConvert.DeserializeObject(Of Dictionary(Of String, Stock))(response)
Example: Live Demo
First thanks for the comments.
I ended up to do it completely different now...
I had further problems with the Ameritrade API:
Some fields are named with leading numbers (52WkHigh and 52WkLow) and
vb.net dev's know, that VB.net don't like properties in classes that
are named with a leading number
So I had to "patch" the received Json data and change the names on the fly to other names ("52WkHigh" to "dble52WkHigh" and "52WkLow" to "dble52WkLow") to be able to deserialize
over the data class, what is not nice
Further, I finally need the data (as fast as possible) in a data table and had "a long way to go":
get data -> deserialize to the data class -> walk thru the data class and overtake the data in the data table.
So.. my new solution (with JObject):
Note: needs:
Imports Newtonsoft.Json.Linq
Code snippets:
Create data table in memory:
Dim dtErgebnis As New DataTable
Dim drTemp As DataRow
With dtErgebnis.Columns
.Add("symbol", System.Type.GetType("System.String"))
.Add("lastPrice", System.Type.GetType("System.Double"))
.Add("lastSize", System.Type.GetType("System.Int32"))
.Add("quoteTime", System.Type.GetType("System.DateTime")) ' Note: is a Long in Json
...
End With
Parse the Json-String and fill the datatable:
get the data over httpclient (in JsonString)...
Dim oJson As JObject = JObject.Parse(JsonString) ' creates children tokens
Dim results As List(Of JToken) = oJson.Children().ToList
For Each item As JProperty In results
item.CreateReader()
drTemp = dtErgebnis.NewRow() ' create a new row to data table in memory
' Fill the fields
drTemp("symbol") = item.Value("symbol")
drTemp("lastPrice") = item.Value("lastPrice")
drTemp("lastSize") = item.Value("lastSize")
drTemp("quoteTime") = GetUTCDateFromTimeStamp(item.Value("quoteTimeInLong")).AddHours(1) ' original Long
...
' Add the new row to the data table
dtErgebnis.Rows.Add(drTemp)
' Save the changes
dtErgebnis.AcceptChanges()
Next
Additional note: The Ameritrade API gives back the time stamps as long (additional hurdle), but I (and I think also you;-) want it as datetime.
Therefore the Long (I think this data type comes from Java/Unix) has to be "translated" to datetime = vb.net function GetUTCDateFromTimeStamp below:
Public Function GetUTCDateFromTimeStamp(TimeStamp As Long) As DateTime
Static startTime As New DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)
Return startTime.AddMilliseconds(TimeStamp)
End Function
Additional note:
As I want to have the Swiss time, I add one hour to the UTC time.
So.. this a real good solution for me (exactly for the Ameritrade API).
And.. it's blazing fast... (I get 19 tickers with all fields and show the result (data table) in a data grid).
All together took < 1 Second ("felt" about 500 ms)
Hope this helps somebody...

Handling Json Array in VB.net

I am getting data from some monitoring equipment which is returning what I believe is a Json (nested) array but the syntax is not quite right.
Below is a sample of what is being retuned. Its an array of UnixTimeMilliseconds and a value.
[[1579617389000,132],[1579617399000,136],[1579617409000,139],[1579617419000,137]]
It could be up to 3000 sets of these numbers.
What I would like to do is dump them into an object of some sort an array so I am able to work with it e.g. translate the UnixTimeMilliseconds into datetime and do calculations.
Last thing I tried was:
Dim ListResultsArray As Directory()
ListResultsArray = JsonConvert.DeserializeObject(Of Directory())(emoncmsResponse)
and the error returned was:
Could not create an instance of type System.IO.Directory. Type is an interface or abstract class and cannot be instantiated. Path '[0].id', line 1, position 7.
A simple method to convert the Arrays is to parse the JSON with JArray, iterate the inner arrays and produce a new class object, converting the array values to the desired type.
Create an object that can represent the array values:
<JsonArray>
Public Class NumbersItem
Public Sub New(objTime As DateTimeOffset, objValue As Long)
DetectionTime = objTime
Value = objValue
End Sub
Public ReadOnly Property DetectionTime As DateTimeOffset
Public ReadOnly Property Value As Long
End Class
Then parse the JSON and generate new class objects from each inner JArray object:
Since the first value in the Array represent an Unix DateTime expressed in milliseconds, we can use the DateTimeOffset.FromUnixTimeMilliseconds() method to convert it.
Dim numbers As New List(Of NumbersItem)()
Dim jsonArrays = JArray.Parse(json)
For Each array As JArray In jsonArrays.Children()
numbers.Add(New NumbersItem(
DateTimeOffset.FromUnixTimeMilliseconds(CType(array.First, Long)),
CType(array.Last, Long)))
Next
Now, you can inspect each NumbersItem class object in the list as usual.
The DetectionTime property is defined as a DateTimeOffset because the Unix time is expressed in UTC coordinates. You can use the DateTimeOffset methods to determine the local DateTime (DateTimeOffset.ToLocalTime) or perform other conversions.
Have you tried JObject using NewtonSoft.json.Linq
Just create a JObject Array and parse Json
Dim object As JObject()
JObject.parse(object(index))

How to deserialize theefold nested json data?

Consuming a RestAPI I get a Json file that holds data of surveys. I load the file into the following class:
Public Class GeneralInfo
Public Property id As Integer
Public Property name As String
Public Property data As List(Of FullData)
End Class
Every survey has an id, a name and data.
Public Class FullData
Public Property userID As ULong
Public Property data As List(Of String)
End Class
Every survey holds data for each participant. I use Newtonsoft.Json to get the data into my classes.
JsonConvert.DeserializeObject(Of List(Of GeneralInfo))(jstr)
The data within the data-property is comma separated but the length is different for every survey. So in survey 1 can be 100 datapoints (one for each answer to a question) and in survey 2 can be up to 400 datapoints. A few of these datapoints can include comma as well since there are datapoints that hold open questions.
I just do not know how to store these data properly. I already read into the JsonConvert and could accomplish getting the data into the classes you can see above but here it ends. Could anyone help me out here? I address these to c# as well since I do not care if the solution is c# or vb.net.
Here are some sample data:
[{"id":1,
"name":"2015 - companyname.dpt",
"data":[{
"userId":364,
"data":["367","90","company","","","Anonymous","User","anonymous#provider.org","","100.100.100.100","14242","undefined","0","7891","2","2","3","1","","","","","","","2015","1","","","","","","","","","","","","","","","","3","3","2","1","3","3","3","4","3","3","4","2","3","3","2","3","2","3","4","3","","4","5","3","4","2","4","4","4","4","4","2","4","4","2","2","4","3","4","4","4","4","3","","1","1","1","5","1","1","1","","1","1","1","1","1","3","3","3","3","4","4","1","","","","2","2","3","1","","","","","","","","",""]
}, {"userID": 365, ...}]
}]

Parsing JSON Objects within JSON arrays within JSON Objects in VB.Net

Hello Everyone and thanks for looking at this. I'm relatively new to vb.net and extremely new to parsing json in vb. I am using JSON.Net and I'm looking to gather data from the following JSON.
http://hastebin.com/bagiyetece.apache
I have classes created for each of the "sections". I am unsure of the correct terminology.
Class One:
Public Class StatsWrapper
Public SummonerID as Long
Public PlayerStatSummaries as playerStatSummaryTypeWrapper
End Class
Class Two:
Public Class playerStatSummaryTypeWrapper
Public playerStatSummaryType As String
Public wins As Long
Public modifyDate As Long
Public aggregatedStats As aggregatedStatsWrapper
End Class
Class Three:
http://hastebin.com/qopanafabe.php
My end goal is to be able to get elements like "totalChampionKills" for the playerStatSummaryType of "Cap5x5" and insert them into a datagridview.
I've been able to correctly parse the following JSON simply by using JObject.Parse.
{"UserName":{"id":84737282,"name":"UserName","profileIconId":660,"summonerLevel":30,"revisionDate":1455686689000}}
To get the id object, I would use:
Dim JSONDerulo = JObject.Parse(JSONResponse)
Dim SummonerID = JSONDerulo(LCase(ToolStripTextBox1.Text))("id")
Where the ToolStripTextBox1.Text is the UserName.
When I try to apply the same logic as above to the larger JSON file in a different sub:
Dim JSONDerulo = JObject.Parse(JSONResponse)
Dim PlayerStatSummaries = JSONDerulo("playerStatSummaries")
Dim Jarray As JArray = PlayerStatSummaries
I can do something like:
For Each obj As JObject In Jarray
TextBox2.Text = TextBox2.Text + Environment.NewLine + obj.ToString + Environment.NewLine
Next
Or call Jarray(1) and parse that again but the list of game types is going to be different for each UserName. I do have a master list for every game type though:
AramUnranked5x5, Ascension, Bilgewater, CAP5x5, CoopVsAI, CoopVsAI3x3, CounterPick, FirstBlood1x1, FirstBlood2x2, Hexakill, KingPoro, NightmareBot, OdinUnranked, OneForAll5x5, RankedPremade3x3, RankedPremade5x5, RankedSolo5x5, RankedTeam3x3, RankedTeam5x5, SummonersRift6x6, Unranked, Unranked3x3, URF, URFBots
If I want to (for example) call AramUnranked5x5.TotalChampionKills or something similar would I have to create a class for each type?
Any suggestions or ideas?
Thanks Again
Note: the json posted is considerably shorter and simpler than the gruesome aggregatedStatsWrapper class linked to depicts! Without data, I cant say whether it is right or not.
Since the aggregatedStats is its own type, nothing very interesting will show for it in a DataGridView, just the type name. There are several ways to handle this. One is to hide the property from the DGV, then when they change selected/current rows, find the new one in the list and set player.aggregatedStats as the selected object in a property grid for a master-detail type view. Playerstatsummary:
Public Class Playerstatsummary
Public Property playerStatSummaryType As String
Public Property wins As Integer
Public Property modifyDate As Long
<Browsable(False)>
Public Property aggregatedStats As Aggregatedstats
Public Property losses As Integer
End Class
<Browsable(False)> will result in the TypeName not being shown in a DGV:
Dim jstr = File.ReadAllText("C:\Temp\myjsonfilename")
Dim jobj = JObject.Parse(jstr)
Dim players = JsonConvert.DeserializeObject(Of List(Of Playerstatsummary))(jobj("playerStatSummaries").ToString)
By Parsing it first, you can skip that outer container. jobj("playerStatSummaries").ToString passes the property data to be deserialized into a List.
You can display what you have very easily without having to loop at all:
dgv1.DataSource = players
It wont yet know about Aggregatedstats unless and until you work out that class exactly. Until then, the type name will display. The post mentions being interested in Cap5x5 only. In that case, a PropertyGrid might be a better UI mechanism (after you find that guy in the list). Result:
(This is from before I added <Browsable(False)> to the class). You could show aggregatedStats as detail like this:
Private Sub DataGridView1_SelectionChanged(...etc
If DataGridView1.SelectedRows.Count = 0 Then Return
Dim thisOne = DataGridView1.SelectedRows(0).Cells(0).Value.ToString
Dim player = players.FirstOrDefault(Function(f) f.playerStatSummaryType = thisOne)
If player IsNot Nothing Then
PropertyGrid1.SelectedObject = player.aggregatedStats
End If
End Sub
In case you are wondering, the date is almost certainly a Unix Epoch date, easily converted to .NET.

Parsing a file containing multiple JSON objects separated by blank lines or tabs

I am having trouble trying to get JSON parsed correctly here. I have the following format and tried using JObjects, but what it does is split one object into its different objects. Perhaps an example will make sense:
{
"completed_in": 0.012,
"max_id": 136536013832069120,
"max_id_str": "136536013832069120",
"next_page": "?page=2&max_id=136536013832069120&q=twitterapi&rpp=1",
"page": 1,
"query": "twitterapi",
"refresh_url": "?since_id=136536013832069120&q=twitterapi",
"results": [
{
"created_at": "Tue, 15 Nov 2011 20:08:17 +0000",
"from_user": "fakekurrik",
"from_user_id": 370773112,
"from_user_id_str": "370773112",
"from_user_name": "fakekurrik",
"geo": null,
"id": 136536013832069120,
"id_str": "136536013832069120",
"iso_language_code": "en",
"metadata": {
"result_type": "recent"
},
"profile_image_url": "http://a1.twimg.com/profile_images/1540298033/phatkicks_normal.jpg",
"source": "<a href="http://twitter.com/">web</a>",
"text": "#twitterapi, keep on keeping it real",
"to_user": "twitterapi",
"to_user_id": 6253282,
"to_user_id_str": "6253282",
"to_user_name": "Twitter API"
}
],
"results_per_page": 1,
"since_id": 0,
"since_id_str": "0"
}
This is what I consider one object. I have files that have hundreds of these and just separated by a tab or blank line. Now if I use JObject
Dim jobj As JObject = JObject.Parse(txtStuff.ToString())
Dim results As List(Of JToken) = jobj.Children().ToList
Results contains all the individual tokens. How can I get each object like the above (the entire object) into a list to process?
It sounds like you're really asking two questions here.
Given the above JSON, how do I get the data into a nice object structure?
Given that I have files containing lots of these objects, how do I get them into a list?
The first part is very easy. Just define a class structure that matches your JSON, then use JsonConvert.DeserializeObject() to deserialize the JSON into that object. For the JSON you posted, the class structure would look something like this:
Class RootObject
Public Property completed_in As Double
Public Property max_id As Long
Public Property max_id_str As String
Public Property next_page As String
Public Property page As Integer
Public Property query As String
Public Property refresh_url As String
Public Property results As List(Of Result)
Public Property results_per_page As Integer
Public Property since_id As Integer
Public Property since_id_str As String
End Class
Class Result
Public Property created_at As String
Public Property from_user As String
Public Property from_user_id As Integer
Public Property from_user_id_str As String
Public Property from_user_name As String
Public Property geo As Object
Public Property id As Long
Public Property id_str As String
Public Property iso_language_code As String
Public Property metadata As Metadata
Public Property profile_image_url As String
Public Property source As String
Public Property text As String
Public Property to_user As String
Public Property to_user_id As Integer
Public Property to_user_id_str As String
Public Property to_user_name As String
End Class
Class Metadata
Public Property result_type As String
End Class
You can deserialize it like this:
Dim obj As String = JsonConvert.DeserializeObject(Of RootObject)(json)
So at this point, obj will contain all the data from one object as you have defined it in your question. Now, you have indicated that you have a file that has many of these JSON objects together separated by a tab or a blank line. You can't just read the whole file in and give it to the JSON parser as one big string because this format isn't valid JSON. (Each individual object is valid JSON of course, but when strung together with tabs or blank line separators, the whole is not valid.) So, you will need to read the file in, line by line (or perhaps character by character) to find the separators and break it up into valid JSON objects that the parser can understand. Each time you find a separator, take all the data that you've read since the last separator and feed that to the deserializer. The result of each deserialization will be a valid RootObject which you can then add to a list as you go along.
Here is some code to give you an idea of how this might work. You may have to tweak it, depending on your needs, but I'm guessing it's not that far off the mark.
'' This function will read a file containing a series of JSON objects separated by
'' some string that is NOT part of the JSON. Could be a blank line or a tab or
'' something else. It will return a list of the deserialized JSON objects.
'' This function relies on two other helper functions (below).
Function ReadJsonFile(fileName As String, separator As String) As List(Of RootObject)
Dim objects As New List(Of RootObject)
Using sr As New StreamReader(fileName)
Dim json As String
Do
json = ReadToSeparator(sr, separator)
If json.Length > 0 Then
objects.Add(JsonConvert.DeserializeObject(Of RootObject)(json))
End If
Loop Until json.Length = 0
End Using
Return objects
End Function
'' This function will read and build up a string until the given separator is found.
'' Once the separator is found, it returns the string with the separator removed.
'' If no separator is found before the end of the data is reached, it returns
'' whatever was read to that point.
Function ReadToSeparator(reader As TextReader, separator As String) As String
Dim sb As New StringBuilder()
While reader.Peek <> -1
Dim ch As Char = ChrW(reader.Read())
sb.Append(ch)
If TailMatchesSeparator(sb, separator) Then
sb.Remove(sb.Length - separator.Length, separator.Length)
Exit While
End If
End While
Return sb.ToString()
End Function
'' This function checks whether the last characters in a StringBuffer match the
'' given separator string. Returns true if so or false if not.
Function TailMatchesSeparator(sb As StringBuilder, separator As String) As Boolean
If sb.Length >= separator.Length Then
Dim i As Integer = sb.Length - 1
For j As Integer = separator.Length - 1 To 0 Step -1
If sb(i) <> separator(j) Then
Return False
End If
i = i - 1
Next
Return True
End If
Return False
End Function
To use this, just call ReadJsonFile, passing a file name and a separator string. For example:
Dim separator As String = vbCrLf + vbCrLf
Dim objects As List(Of RootObject) = ReadJsonFile("json.txt", separator)