EF Core 7 can't deserialize dynamic-members in JSON column - json

I am trying to map my Name column to a dynamic object. This is how the raw JSON data looks (note that this is SQL-morphed from our old relational data and I am not able to generate or interact with this column via EF Core):
{ "en": "Water", "fa": "آب", "ja": "水", ... }
Just to note, available languages are stored in a separate table and thus are dynamically defined.
Through T-SQL I can perfectly interact with these objects eg
SELECT *
FROM [MyObjects]
WHERE JSON_VALUE(Name, '$.' + #languageCode) = #searchQuery
But it seems EF Core doesn't want to even deserialize these objects as whole, let alone query them.
What I get in a simple GetAll query is an empty Name. Other columns are not affected though.
I have tried so far
Using an empty class with a [JsonExtensionData] dictionary inside
Using a : DynamicObject inheritance and implementing GetDynamicMembers, TryGetMember, TrySetMember, TryCreateInstance
Directly mapping to a string dictionary.
Combining 1 & 2 and adding an indexer operator on top.
All yield the same results: an empty Name.
I have other options like going back to a junction table relational which I have many issues with, hardcoding languages which is not really intuitive and might cause problems in the future, using HasJsonConversion which basically destroys the performance on any search action... so I'm basically stuck here with this.

I think currently it's not fully supported:
You can not use dynamic operations on an expression tree like a Select statement because it needs to be translated.
JsonValue and JsonQuery requires a path to be resolved.
If you specify OwnsOne(entity = >entity.owned, owned => owned.ToJson()) and the Json could not be parsed you will get an error.
I suggest this workaround while the EF team improves the functionality.
Create a static class with static methods to be used as decoys in the expression tree. This will be mapped to the server built-in functions.
public static class DBF
{
public static string JsonValue(this string column, [NotParameterized] string path)
=> throw new NotSupportedException();
public static string JsonQuery(this string column, [NotParameterized] string path) => throw new NotSupportedException();
}
Include the database functions on your OnModelCreating method.
protected override void OnModelCreating(ModelBuilder modelBuilder)
{
base.OnModelCreating(modelBuilder);
modelBuilder.HasDbFunction(
typeof(DBF).GetMethod(nameof(DBF.JsonValue))!
).HasName("JSON_VALUE").IsBuiltIn();
modelBuilder.HasDbFunction(
typeof(DBF).GetMethod(nameof(DBF.JsonQuery))!
).HasName("JSON_QUERY").IsBuiltIn();
/// ...
modelBuilder.Entity(entity => {
//treat entity as text
entity.Property(x => x.Metadata)
.HasColumnType("varchar")
.HasMaxLength(8000);
});
}
Call them dynamically with LINQ.
var a = await _context.FileInformation
.AsNoTracking()
.Where(x => x.Metadata!.JsonValue("$.Property1") == "some value")
.Select(x => x.Metadata!.JsonValue("$.Property2"))
.ToListAsync();
You can add casts or even build anonymous types with this method.

My solution was I added a new class which has KEY and VALUE , which will represent the dictionary i needed :
public class DictionaryObject
{
public string Key { set; get; }
public string Value { set; get; }
}
and instead of having this line in the JSON class :
public Dictionary<string, string> Name { get; set; }
I changed to :
public List<DictionaryObject> Name { get; set; }
Hope it helps.

Related

How to use ServiceStack to store POCOs to MariaDB having complex types (objects and structs) blobbed as JSON?

I've got following setup: C#, ServiceStack, MariaDB, POCOs with objects and structs, JSON.
The main question is: how to use ServiceStack to store POCOs to MariaDB having complex types (objects and structs) blobbed as JSON and still have working de/serialization of the same POCOs? All of these single tasks are supported, but I had problems when all put together mainly because of structs.
... finally during writing this I found some solution and it may look like I answered my own question, but I still would like to know the answer from more skilled people, because the solution I found is a little bit complicated, I think. Details and two subquestions arise later in the context.
Sorry for the length and for possible misinformation caused by my limited knowledge.
Simple example. This is the final working one I ended with. At the beginning there were no SomeStruct.ToString()/Parse() methods and no JsConfig settings.
using Newtonsoft.Json;
using ServiceStack;
using ServiceStack.DataAnnotations;
using ServiceStack.OrmLite;
using ServiceStack.Text;
using System.Diagnostics;
namespace Test
{
public class MainObject
{
public int Id { get; set; }
public string StringProp { get; set; }
public SomeObject ObjectProp { get; set; }
public SomeStruct StructProp { get; set; }
}
public class SomeObject
{
public string StringProp { get; set; }
}
public struct SomeStruct
{
public string StringProp { get; set; }
public override string ToString()
{
// Unable to use .ToJson() here (ServiceStack does not serialize structs).
// Unable to use ServiceStack's JSON.stringify here because it just takes ToString() => stack overflow.
// => Therefore Newtonsoft.Json used.
var serializedStruct = JsonConvert.SerializeObject(this);
return serializedStruct;
}
public static SomeStruct Parse(string json)
{
// This method behaves differently for just deserialization or when part of Save().
// Details in the text.
// After playing with different options of altering the json input I ended with just taking what comes.
// After all it is not necessary, but maybe useful in other situations.
var structItem = JsonConvert.DeserializeObject<SomeStruct>(json);
return structItem;
}
}
internal class ServiceStackMariaDbStructTest
{
private readonly MainObject _mainObject = new MainObject
{
ObjectProp = new SomeObject { StringProp = "SomeObject's String" },
StringProp = "MainObject's String",
StructProp = new SomeStruct { StringProp = "SomeStruct's String" }
};
public ServiceStackMariaDbStructTest()
{
// This one line is needed to store complex types as blobbed JSON in MariaDB.
MySqlDialect.Provider.StringSerializer = new JsonStringSerializer();
JsConfig<SomeStruct>.RawSerializeFn = someStruct => JsonConvert.SerializeObject(someStruct);
JsConfig<SomeStruct>.RawDeserializeFn = json => JsonConvert.DeserializeObject<SomeStruct>(json);
}
public void Test_Serialization()
{
try
{
var json = _mainObject.ToJson();
if (!string.IsNullOrEmpty(json))
{
var objBack = json.FromJson<MainObject>();
}
}
catch (System.Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
public void Test_Save()
{
var cs = "ConnectionStringToMariaDB";
var dbf = new OrmLiteConnectionFactory(cs, MySqlDialect.Provider);
using var db = dbf.OpenDbConnection();
db.DropAndCreateTable<MainObject>();
try
{
db.Save(_mainObject);
var dbObject = db.SingleById<MainObject>(_mainObject.Id);
}
catch (System.Exception ex)
{
Debug.WriteLine(ex.Message);
}
}
}
}
What (I think) I know / have tried but at first didn't help to solve it myself:
ServiceStack stores complex types in DB as blobbed JSV by default (last paragraph of first section: https://github.com/ServiceStack/ServiceStack.OrmLite), so it is necessary to set it the way it is proposed: MySqlDialect.Provider.StringSerializer = new JsonStringSerializer(); (https://github.com/ServiceStack/ServiceStack.OrmLite#pluggable-complex-type-serializers)=> default JSV changed to JSON.
the ServiceStack's serialization does not work with structs, it is necessary to treat them special way:
a) according to https://github.com/ServiceStack/ServiceStack.Text#c-structs-and-value-types and example https://github.com/ServiceStack/ServiceStack.Text/#using-structs-to-customize-json it is necessary to implement TStruct.ToString() and static TStruct.ParseJson()/ParseJsv() methods.
b) according to https://github.com/ServiceStack/ServiceStack.Text/#typeserializer-details-jsv-format and unit tests https://github.com/ServiceStack/ServiceStack.Text/blob/master/tests/ServiceStack.Text.Tests/CustomStructTests.cs it shall be TStruct.ToString() (the same as in a) and static TStruct.Parse().
Subquestion #1: which one is the right one? For me, ParseJson() was never called, Parse() was. Documentation issue or is it used in other situation?
I implemented option b). Results:
IDbConnection.Save(_mainObject) saved the item to MariaDB. Success.
Through the saving process ToString() and Parse() were called. In Parse, incoming JSON looked this way:
"{\"StringProp\":\"SomeStruct's String\"}". Fine.
Serialization worked. Success.
Deserialization failed. I don't know the reason, but JSON incoming to Parse() was "double-escaped":
"{\\\"StringProp\\\":\\\"SomeStruct's String\\\"}"
Subquestion #2: Why the "double-escaping" in Parse on deserialization?
I tried to solve structs with JsConfig (and Newtonsoft.Json to get proper JSON):
JsConfig<SomeStruct>.SerializeFn = someStruct => JsonConvert.SerializeObject(someStruct);
JsConfig<SomeStruct>.DeSerializeFn = json => JsonConvert.DeserializeObject<SomeStruct>(json);
a) at first without ToString() and Parse() defined in the TStruct. Results:
Save failed: the json input in JsonConvert.DeserializeObject(json) that is used during Save was just type name "WinAmbPrototype.SomeStruct".
De/serialization worked.
b) then I implemented ToString() also using Newtonsoft.Json. During Save ToString() was used instead of JsConfig.SerializeFn even the JsConfig.SerializeFn was still set (maybe by design, I do not judge). Results:
Save failed: but the json input of DeserializeFn called during Save changed, now it was JSV-like "{StringProp:SomeStruct's String}", but still not deserializable as JSON.
De/serialization worked.
Then (during writing this I was still without any solution) I found JsConfig.Raw* "overrides" and tried them:
JsConfig<SomeStruct>.RawSerializeFn = someStruct => JsonConvert.SerializeObject(someStruct);
JsConfig<SomeStruct>.RawDeserializeFn = json => JsonConvert.DeserializeObject<SomeStruct>(json);
a) at first without ToString() and Parse() defined in the TStruct. Results are the same as in 2a.
b) then I implemented ToString(). Results:
BOTH WORKED. No Parse() method needed for this task.
But it is very fragile setup:
if I removed ToString(), it failed (now I understand why, default ToString produced JSON with just type name in 2a, 3a).
if I removed RawSerializeFn setting, it failed in RawDeserializeFn ("double-escaped" JSON).
Is there some simpler solution? I would be very glad if someone points me to better direction.
Acceptable would be maybe two (both of them accessible because of different circumstances):
if I am the TStruct owner: with just pure TStruct.ToString() and static TStruct.Parse() to support out of the box de/serialization and DB by ServiceStack (without different input in Parse()).
if I am a consumer of TStruct with no JSON support implemented and I am without access to its code: until now I did not find the way, if the ToString is not implemented: Save to DB did not work. Maybe would be fine to ensure JsConfig serialize functions are enough for both de/serialization and when used during saving to DB.
And the best one would be without employing other dependency (e.g. Newtonsoft.Json) to serialize structs. Maybe some JsConfig.ShallProcessStructs = true; (WARNING: just a tip, not working as of 2021-04-02) would be fine for such situations.
ServiceStack treats structs like a single scalar value type, just like most of the core BCL Value Types (e.g. TimeSpan, DateTime, etc). Overloading the Parse() and ToString() methods and Struct's Constructor let you control the serialization/deserialization of custom structs.
Docs have been corrected. Structs use Parse whilst classes use ParseJson/ParseJsv
If you want to serialize a models properties I'd suggest you use a class instead as the behavior you're looking for is that of a POCO DTO.
If you want to have structs serailized as DTOs in your RDBMS an alternative you can try is to just use JSON.NET for the complex type serialization, e.g:
public class JsonNetStringSerializer : IStringSerializer
{
public To DeserializeFromString<To>(string serializedText) =>
JsonConvert.DeserializeObject<To>(serializedText);
public object DeserializeFromString(string serializedText, Type type) =>
JsonConvert.DeserializeObject(serializedText, type);
public string SerializeToString<TFrom>(TFrom from) =>
JsonConvert.SerializeObject(from);
}
MySqlDialect.Provider.StringSerializer = new JsonNetStringSerializer();

Returning a Json-field from SQL to ASP.NET Core API

I'm building a relatively simple Get-method in an ASP.NET Core (3+) application. (Currently 3.1 - to be migrated to 5)
The object I need to return looks like this:
public class Data
{
public int ID { get;set;}
public string Name { get;set;}
public string Settings { get; set;}
}
And the Get-method is simply this:
public IActionResult<Data> GetData()
{
var data = _dbContext.GetData<Data>();
return Ok(data);
}
This works perfectly - except for one thing.
In SQL - the settings column (varchar(8000)), contains JSON data. In some cases, a setting can be something simple like : { "threshold": 8754 } and sometimes it can be a large complex object with many fields, but it is always valid Json.
On the ASP side, it does exactly what you would expect. It turns a serialized Json object that contains an INT and 2 x strings.
I would like for it to return an INT, ONE String and One Json Object.
Is there any way that I can tell the serializer that the Settings-property contains Json?
In a perfect world, I would love something like this:
public class Data
{
public int ID { get;set;}
public string Name { get;set;}
[SerializeContentAsJson]
public string Settings { get; set;}
}
Is there a way to do this or is there some other fairly elegant solution to this problem?
Btw. I fully realize that the caller can specify the content types that he/she will accept. In this case, the API is purely for use inside my team and we will always want JSON, so I can compromise on this being a relatively custom solution that might not work if you wanted text/html or some other content type.

How would you get this data in unity

{"country":{"code":"NZ"}}
How would you store this data in a unity class?
Something like this:
using UnityEngine;
using System;
using System.Collections;
using System.Globalization;
[Serializable]
public class GDRPClass
{
public string country;
public string code;
}
I dont think that is right for a double {} though.
Thanks
Always a very good starting point: json2csharp!
just make sure to use fields instead of properties and mark the clases [Serializable]:
[Serializable]
public class Country
{
public string code;
}
[Serializeable]
public class GDRPClass
{
public Country country;
}
You can call the classes/structs whatever you like, but the field names have to match!
In this simple case the structure is easily explained since the data types in JSON are very limited. You can see it better if you use the intended notation
{
"country" :
{
"code" : "NZ"
}
}
{ } in json always wrap a json object (= class/struct). So you already know there have to be exactly two class/struct types in your case.
you can see the first (root) type has to have one field called country. This field's type is the other class/struct's type.
this second inner type has another field called code. It's type is a string.
And that's exactly what json2csharp already spit out for us ;)
Now you can e.g. use JsonUtility.FromJson
var json = "{\"country\":{\"code\":\"NZ\"}}";
var jsonObject = JsonUtility.FromJson<GDRPClass>(json);
and later access the data using e.g.
var code = jsonObject.country.code; // = "NZ"
"country" is basically used as a key for "{"code":"NZ"}", so they're on different levels of the tree.
You could deserialize this into a Dictionary<string, CountryInfo> (but not with with the built-in JsonUtility, that doesn't do Dictionaries, so I used Json .Net instead)
Have a look:
using System;
using System.Collections.Generic;
using Newtonsoft.Json;
using UnityEngine;
public class TestDeserialize : MonoBehaviour {
[SerializeField] private string m_JsonStr = "{\"New Zealand\":{\"code\":\"NZ\"}}";
void Start() {
Dictionary<string,CountryInfo> result = JsonConvert.DeserializeObject<Dictionary<string, CountryInfo>>(m_JsonStr);
foreach (KeyValuePair<string,CountryInfo> country in result) { //Log results
Debug.LogFormat("{0}: {1}", country.Key, country.Value.code);
}
Debug.Log(JsonConvert.SerializeObject(result)); //Serializes back into its original form
}
[Serializable]
class CountryInfo {
public string code;
}
}
I use Pathfinding.Serialization.JsonFx for JSON Parsing
{\"country\":4,\"code\":\"Afghanistan\"}
This will be the jsonString with respect to your Model class
GDRPClass data = JsonReader.Deserialize<GDRPClass>JsonWriter.Serialize(jsonString));

Converting simple value to JSON in ASP.NET Core API

Sometimes my ASP.NET Core API needs to return a simple value i.e. bool, int or string even though in most cases, I return complex objects/arrays as JSON.
I think for consistency purposes, it's a good idea to return even simple values as JSON. What's the easiest way to convert a simple value, whether it's bool or int into JSON?
My standard controller action looks like this -- see below -- which gives me the ability to return status codes as well as data. Therefore, I'd like to stick to that approach, rather than return JsonResult.
public async Task<IActionResult> Get()
{
// Some logic
return Ok(data);
}
I'm just trying to figure out the easiest way to convert my data into JSON, if it's not already in JSON format.
Looking at your code, I assume your application is supposed to be a service that needs to return some kind of data serialised in JSON.
Well, good news is ASP.NET Core already includes a data serialiser that would do the job for you.
You may need to set it up according to your needs.
For example, let's assume the following data class:
public class Data {
public string Name { get; }
public string Value { get; }
public bool IsValid { get; }
public Data(string name, string value, bool isValid) {
Name = name;
Value = value;
IsValid = isValid;
}
}
Then the following method in your Controller:
public async Task<IActionResult> Get() {
var data = new Data("sample name", "this is a value", true);
return Ok(data);
}
would return:
{
"name": "sample name",
"value": "this is a value",
"isValid": true
}
Even thought the standard serialisation behaviour may fit fine for very simple implementations, you may need more control on how your different data types should be serialised (and deserialised) by your application, especially when those do not exactly match the way you want to present the data back to the client. In this case you may want to use Custom Converters.
You can configure that when setting up MVC in the ConfigureServices(IServiceCollection services) method:
// Add framework services.
services.AddMvc().AddJsonOptions(jo => {
// sample serialiser setup
jo.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();
jo.SerializerSettings.DateTimeZoneHandling = DateTimeZoneHandling.Utc;
jo.SerializerSettings.MissingMemberHandling = MissingMemberHandling.Error;
// custom Converters
jo.SerializerSettings.Converters.Add(new MyCustomConverter());
});
Here you can read and learn more on how to setup and use Custom Converters.

Can the GXT JsonReader (using AutoBeans) parse complex JSON structures in one request?

I need to work with data returned from a service which has a more complex JSON structure than the examples provided in the GXT documentation and thus far I cannot find any instructions or example which demonstrates how this might be accomplished.
The JSON contains multiple key/value pairs, but some of the key/value pairs are collections. I can have all of the data returned to me in one call from the service in the proper structure, but there does not appear to be a way to parse the data into separate entities. In my particular case I am attempting to configure a loader which will process one of the collections but I also need other key/value pairs from the same message (it is not ok to have the loader make one call and then have another call made for the same data and retrieve the other key/value pairs). Is there any way to accomplish this using GXT3?
Example: let's assume I can make a request from a server which returns JSON containing the name of an author along with a collection of the books the author has written. I want to display the author's name above a grid which lists the books. I want only one request made to the server and then have my view display the author in one component and the book list in a grid. Assume I need a loader instead of just a store as the grid may have to make additional calls (e.g. if it is a paging grid, livegrid, etc.).
Example JSON: (one JSON message returned with and author element along with a collection of book elements - I've indented the JSON to illustrate the structure)
{ "returnData" :
{"author" : "AuthorName"},
{"books" :
{"id" : "1", "name" : "Book1"},{"id" : "2", "name" : "Book2"}
}
}
Using the example for JsonReader (see the javadoc for an example) I can receive the request and parse the links into a collection using AutoBeans. This works fine when I need to have those retrieved and parsed in a loader. However, if I do that then the other properties are ignored. I currently don't see any way to parse the other values in the same request so they can be used elsewhere. My example code for the collection processing is below:
// this is the root JSON object, the AuthorRecord
public interface AuthorRecord {
#PropertyName(value="author")
String getAuthor();
#PropertyName(value="author")
void setAuthor(String author);
#PropertyName(value="books")
List<Book> getBooks();#
#PropertyName(value="books")
void setBooks (List<Book> books);
}
// models the book objects returned
public interface Book {
#PropertyName(value="id")
String getId();
#PropertyName(value="id")
void setId(String id);
#PropertyName(value="name")
String getName();
#PropertyName(value="name")
void setName(String name);
}
public interface ReturnData {
AuthorRootObject getAuthorRoot();
}
public interface LibraryAutoBeanFactory extends AutoBeanFactory {
AutoBean<ReturnData> authorRecord();
AutoBean<ListLoadConfig> loadConfig();
}
public class ReturnDataJsonReader extends JsonReader<ListLoadResult<Book>,
ReturnData> {
public ReturnDataJsonReader(AutoBeanFactory factory,
Class<ReturnData> rootBeanType) {
super(factory, rootBeanType);
}
#Override
protected ListLoadResultBean<Book> createReturnData(Object loadConfig,
ReturnData incomingData) {
return new ListLoadResultBean<Book>(incomingData.getBooks());
}
}
The problem I was having was that I need to have a view that includes a grid (paging grid, etc.) which lists out the books, while having the Author's name sit above the grid. I wanted to get all of this information (or at least the first page of results) with only one request to the server since the JSON message contains all the information I need to accomplish this. The problem is that the loader makes the request and receives the response, and it expects that the reader it will use is going to process a collection. In my case, I need the loader to process the collection of books but also populate another data field. The solution I found was to create an arbitrary collection to pass to the loader and then implement my own load handler to process the return object as needed.
1.The JSON being returned is really just one object of type ReturnData. The extended JsonReader could process this using AutoBeans, but if the reader is to be used for the loader, it needs to return a collection. Therefore, override the createReturnData() method to return a collection of one object.
public class ReturnDataJsonReader extends JsonReader<ListLoadResult<AuthorRecord>,
ReturnData> {
public ReturnDataJsonReader(AutoBeanFactory factory, Class<ReturnData> rootBeanType)
{
super(factory, rootBeanType);
}
#Override
protected ListLoadResultBean<AuthorRecord> createReturnData(Object loadConfig,
ReturnData incomingData) {
List<AuthorRecord> authorDataCollection = new ArrayList<AuthorRecord>();
authorDataCollection.add(incomingData);
return authorDataCollection;
}
}
2.The LoadHandler used in the examples takes a ListStore as an input and populates it with the results from the loader. Since the return object is not what we want populating the loader, and since we need to populate another property on the view, create your own LoadHandler to take the objects needed as input and populate them:
View Class Example:
public class ExampleViewClass {
// truncating most of the code in here for brevity
// note some of the objects referenced here reference objects in the question
private String authorName;
private ListStore<Book> bookList;
// IMPORTANT - create your own LoadHandler
private class LibraryLoadResultistStoreBinding<C, M, D extends
ListLoadResult<AuthorRecord>> implements LoadHandler<ListLoadConfig,
ListLoadResult<AuthorRecord>> {
private final ListStore<Book> bookStore;
private final String authorName;
public LibraryLoadResultistStoreBinding(ListStore<Book> books, String author) {
this.bookStore = books;
this.authorName = author;
}
#Override
public void onLoad(LoadEvent<ListLoadConfig, ListLoadResult<AuthorRecord> event)
{
// the response object
AuthorRecord response = event.getLoadResult().getData().get(0);
bookStore.replaceAll(response.getBooks());
author = response.getAuthor();
}
}
// example uses an HttpProxy but that's not required
public void populateView() {
LibraryAutoBeanFactory factory = GWT.create(LibraryAutoBeanFactory.class);
ReturnDataJsonReader reader = new ReturnDataJsonReader(factory, ReturnData.class);
String path = "http://path.to.resource/getinfo";
RequestBuilder builder = new RequestBuilder(RequestBuilder.GET, path);
HttpProxy<ListLoadConfig> proxy = new HttpProxy<ListLoadConfig>(builder);
final ListLoader<ListLoadConfig, ListLoadResult<AuthorRecord>> loader = new
ListLoader<ListLoadConfig, ListLoadResult<AuthorRecord>> (proxy, reader);
loader.useLoadConfig(ReturnDataAutoBeanFactory.instance.loadConfig().as();
loader.addLoadHandler(new LibraryLoadResultistStoreBinding<ListLoadConfig,
AuthorRecord, ListLoadResult<AuthorRecord>>(bookList, authorName);
// pass in the objects to be populated
loader.load(); // fire the loader
}