I have a column defined as follows:
#Type(type = "json")
#Column(name = "ex_data")
#JsonProperty
#JsonTypeInfo(use = JsonTypeInfo.Id.CLASS)
private T exData;
Where, given type T, I can store a variety of basic JSON objects in my table, like so:
{
"active": true,
"color": "red",
"flavor": "cherry"
}
If I want to created a sorted, paginated, query against the table, is it possible to use the content of the JSON for the Sort object?
i.e. an equivalent to this, which presently does not work:
Sort sort = Sort.by("exData.color").ascending();
JSON is not directly supported by JPA.
One way would be to use a native SQL function to access a specific property nested inside the json column.
Unfortunately, native expressions are not supported when using Sort, which only accepts entity properties to order by.
Solution 1: native query with json key parameter and pagination
#Repository
public interface JsonSortNativeRepositoryMySql extends JpaRepository<JsonSortNativeEntity, Long> {
#Query(value = "SELECT * FROM json_sort_native_entity ORDER BY "
// flexibly extract the sort value from json:
+ "my_json_column->> :sortKey" //
+ " DESC", //
countQuery = "select count(*) from json_sort_native_entity", //
nativeQuery = true)
Page<JsonSortNativeEntity> findSortedNative(Pageable pageable, #Param("sortKey") String sortKey);
}
The corresponding pageable is created without Sort parameter, and the sort key can be dynamically supplied when calling the native query method:
var pageable = PageRequest.of(0, 20); // first page with 20 entries
var pageResult = repository.findSortedNative(pageable, "$.color");
Solution 2: sort by computed attribute value
Another option would be to specify a computed attribute value using a Hibernate #Formula as part of the Entity:
#Formula("my_json_column->>'$.color'") // <-- fixed key "color" to extract from json column
private String formulaValue;
This way, a normal Sort can be used:
var pageable = PageRequest.of(0, 20, Sort.by("formulaValue").descending());
var pageResult = repository.findAll(pageable); // standard JpaRepository method
However, the key to extract the sort values is fixed in this case (possible alternative: Spring Data - Page Request - order by function)
MySQL JSON reference: https://dev.mysql.com/doc/refman/8.0/en/json.html
Related
Given there is a dataset of messages, defined by following code:
case class Message(id: Int, value: String)
var messages = Seq(
(0, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}"""),
(1, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}""")
).toDF("id", "value").as[Message]
var schema = new StructType().add("action", StringType).add("timestamp", TimestampType)
var res = messages.select(
from_json(col("value").cast("string"), schema)
)
+------------------------------------+
|jsontostructs(CAST(value AS STRING))|
+------------------------------------+
| [update,2017-10-0...|
| [update,2017-10-0...|
What is the best way to access the schema information in a plain map function. The function itself returns a row which has lost all the Type infos. In order to reach to the values one has to specify the type again e.g
res.head().getStruct(0).getValuesMap[TimestampType](Seq("timestamp"))
=> Map[String,org.apache.spark.sql.types.TimestampType] = Map(timestamp -> 2017-10-06 01:01:19.0)
or
res.head().getStruct(0).getString(0)
=> res20: String = update
Is there some better way to access the raw json data without spark sql aggregation functions?
As a rule of thumb:
To use collection API (map, flatMap, mapPartitions, groupByKey, etc.) use strongly typed API - define record type (case class works the best) which reflects the schema and use Encoders to convert things back and forth:
case class Value(action: String, timestamp: java.sql.Timestamp)
case class ParsedMessage(id: Int, value: Option[Value])
messages.select(
$"id", from_json(col("value").cast("string"), schema).alias("value")
).as[ParsedMessage].map(???)
With Dataset[Row] stay with high level SQL / DataFrame API (select, where, agg, groupBy)
I'm using the Jackson Object Mapper (com.fasterxml.jackson.databind.ObjectMapper) to parse a json string and need to count the number of times the word "path"
occures in the string. The string looks like:
"rows":[{"path":{"uid":"2"},"fields":[]},{"path":{"uid":"4"},"fields":[]},{"path":{"uid":"12"},....
Does anyone know which API option is most efficeint for achieving this?
To count total number of 'childs' inside the 'rows' root, you can use a code like this:
String inputJsonString = "{\"rows\":[{\"path\":{\"uid\":\"2\"},\"fields\":[]},{\"path\":{\"uid\":\"4\"},\"fields\":[]},{\"path\":{\"uid\":\"12\"},\"fields\":[]}]}";
ObjectNode root = (ObjectNode) new ObjectMapper().readTree( inputJsonString );
root.get( "rows" ).size();
If you need to get the exact count of 'path' occurrences, you may use the code like this:
int counter = 0;
for( Iterator<JsonNode> i = root.get( "rows" ).iterator(); i.hasNext(); )
if( i.next().has( "path" ) )
counter++;
System.out.println(counter);
I want to implement a function that accepts a DbSet (non-generic), a string, and object, and returns DbSet. something like the following pseudu:
public static DbSet Any(DbSet set, string propertyName, objectParameter)
{
var tableName = set.TableName;
var columnName = set.GetColumnNameForProperty(propertyName);
var query = string.Format("SELECT TOP(1) {0} FROM {1} WHERE {0} = {2}",
columnName,
tableName,
objectParameter);
}
I think that SQL query is enough since I'll be able to execute it directly on the Database (context.Database.ExecuteSql).
What I want to do is get the table name from the given DbSet, then the column name in the database.
It is not possible from non generic DbSet but this problem can be easily solved by using:
public static IEnumerable<T> Any(DbSet<T> set, string property, objectParameter)
where T : class
{ ... }
Returning DbSet doesn't make sense because once you query data it is not DbSet anymore.
The bigger problem is getting table name from generic DbSet / ObjectSet because this information is not available from those classes. It is almost impossible to get it at all because it requires accessing non public members of items from MetadataWorkspace.
I have Ienumerable<string> collection that I want to concatenate into a single string with delimitor ;
for instance {"One","Two","Three"} -> "One;Two;Three;"
is it possible to do using the following function?
List<string> list = new List<string>(){"One","Two","Three"};
list.Aggregate<String>((x,y) => x + String.Format("{0};",y));
I have tried also this code:
list.Aggregate<String>((x,y) => String.Format("{0};{1}",x,y));
both samples didn't work.
EDIT: I see that it is not possible to do what I wanted using Linq-2-sql or Aggregate function in Linq-2-sql.
http://social.msdn.microsoft.com/forums/en-US/linqprojectgeneral/thread/dac496c0-5b37-43ba-a499-bb8eff178706/
EDIT2: the workaround I used is to go over the items returned by the original linq query...and copies them to a new list and do the join as suggested in the answers below on a linq object and not linq-2-sql object.
You can just use String.Join for this. If you're using .NET4 then you can use the overload that takes an IEnumerable<string> directly:
string joined = string.Join(";", list);
If you're using an older version of the framework then you'll need to use the overload that takes a string[] array instead, converting your collection to an array first if necessary:
string joined = string.Join(";", list.ToArray());
EDIT...
Of course, if you really want to use Aggregate for some reason then there's nothing stopping you. If so, it's usually recommended to build your string using a StringBuilder rather than multiple string allocations:
string joined = list.Aggregate(new StringBuilder(),
(sb, s) => sb.Append(s).Append(';'),
sb => (sb.Length > 0) ? sb.ToString(0, sb.Length - 1)
: "");
You can do it using below code
list.Aggregate((i, j) => i + ";" + j);
You'll need to provide an initializer, otherwise the first element will not have a ; added to it:
list.Aggregate<String>("", (x,y) => x + String.Format("{0};",y));
I would like to query a table based on a list of KeyValuePair. With a Model-First approach, I could do the following:
var context = new DataContext();
var whereClause = new StringBuilder();
var objectParameters = new List<ObjectParameter>();
foreach(KeyValuePair<string, object> pair in queryParameters)
{
if (whereClause.Length > 0)
whereClause.Append(" AND ");
whereClause.Append(string.Format("it.[{0}] = #{0}", pair.Key));
parameters.Add(new ObjectParameter(pair.Key, pair.Value));
}
var result = context.Nodes.Where(whereClause.ToString(), parameters.ToArray());
Now I'm using a Code-First approach and this Where method is not available anymore. Fortunately, I saw an article somewhere (I can't remember anymore) which suggested that I could convert the DbContext to a IObjectContextAdapter then call CreateQuery like this:
var result = ((IObjectContextAdapter)context)
.ObjectContext.CreateQuery<Node>(whereClause.ToString(), parameters.ToArray());
Unfortunately, this throws an error:
'{ColumnName}' could not be resolved in the current scope or context. Make sure that all referenced variables are in scope, that required schemas are loaded, and that namespaces are referenced correctly.
Where {ColumnName} is the column specified in the whereClause.
Any ideas how I can dynamically query a DbSet given a list of key/value pairs? All help will be greatly appreciated.
I think your very first problem is that in the first example you are using Where on the entity set but in the second example you are using CreateQuery so you must pass full ESQL query and not only where clause! Try something like:
...
.CreateQuery<Node>("SELECT VALUE it FROM ContextName.Nodes AS it WHERE " + yourWhere)
The most problematic is full entity set name in FROM part. I think it is defined as name of the context class and name of the DbSet exposed on the context. Another way to do it is creating ObjectSet:
...
.ObjectContext.CreateObjectSet<Node>().Where(yourWhere)