I have this months array:
["January", "March", "December" , "October" ]
And I want to have it sorted like this:
["January", "March", "October", "December" ]
I'm currently thinking in a "if/else" horrible cascade but I wonder if there is some other way to do this.
The bad part is that I need to do this only with "string" ( that is, without using Date object or anything like that )
What would be a good approach?
If I had a way to supply a custom sorting order, I'd create a list defining the correct order:
correct = List("January", "February", "March", ...)
And then sort by the position in that list, something like:
toSort.sort(a, b) => compare(correct.index(a), correct.index(b))
Create a table with name->index, then sort the array based on its value in the table.
A couple of examples may be useful, in C# arr.sort(myCompare), in Java Collections.sort(arr, myCompare), in Python arr.sort(myCompare), in PHP usort($arr, 'myCompare'), in C++ sort(vec.begin(), vec.end(), myCompare).
Have an array with the proper sort, and sort based on it.
Another solution (if your language supports it) is to have an associative array from month names to numbers (1..12) and use a custom comparator running sort on your array.
Solution in Perl :D
my #mon = qw( January February March April May June July August September October November December );
my $mon;
#{$mon}{#mon} = (0..$#mon);
sub by_month {
$mon->{$a} <=> $mon->{$b};
}
sort by_month #data_to_sort
(although I'm sure a golfer could do that in < 30 characters)
And here's a solution in plain C : http://www.pnambic.com/CPS/SortAnal/html/MonOrder.html
Talking from a Java POV, I'm going to pimp (as I often do) google-collections (soon to be replaced with Guava):
Arrays.sort(myMonths, Ordering.explicit("Jan", "Feb", "Mar", ....));
... and you're done.
Don't write it yourself if someone else has done it, probably more efficiently and with a nicer API than you probably would.
Not helpful in the general case, but just in case any Java folk have the same problem...
Colleagues,
I see the issue/business problem lasts more than 2 year.
I decided to write comparator for sorting months' names (stored as strings) properly.
It also holds names of the months for desired locale
============== Comparator =======================
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Comparator;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
/**
*
* #author akashtalyan
*/
public class MonthNamesComparator implements Comparator {
private static Map<Locale, List> monthMap = new HashMap<Locale, List>();
private final Locale locale;
public MonthNamesComparator(Locale locale) {
if (locale == null) {
throw new NullPointerException("MonthNamesComparator cannot accept null value for Locale parameter.");
}
List months = new ArrayList(12);
Calendar cal = Calendar.getInstance(locale);
SimpleDateFormat dateFormat = new SimpleDateFormat("MMMM", locale);
this.locale = locale;
if (!monthMap.containsKey(locale)) {
for (int i = 0; i < 12; i++) {
cal.set(Calendar.MONTH, i);
months.add(dateFormat.format(cal.getTime()).toLowerCase());
}
monthMap.put(locale , months);
}
}
#Override
public int compare(Object month1, Object month2) {
List months = monthMap.get(this.locale);
if (months == null) {
throw new NullPointerException("MonthNamesComparator cannot perform comparison - internal data is not initialized properly.");
}
return (months.indexOf(((String) month1).toLowerCase()) - months.indexOf(((String) month2).toLowerCase()));
}
}
And simple test class to POC:
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
/**
*
* #author akashtalyan
*/
public class TestMonths {
public static void main(String[] args){
Locale en = Locale.ENGLISH, ru = new Locale("ru","RU");
String[] monthsToTestEn = new String[] {"FebRUary", "maY", "sepTember", "january", "december"};
String[] monthsToTestRu = new String[] {"АпреЛь", "январь", "Март", "Август"};
Map map = new TreeMap(new MonthNamesComparator(en));
int i = 0;
System.out.println("En Map original:");
for (String month : monthsToTestEn) {
System.out.print(month + " ");
map.put(month, new StringBuilder(String.valueOf(++i)).append(" position in source array").toString());
}
System.out.println();
System.out.println("En Map sorted:");
for (String month : (Set<String>)map.keySet()) {
System.out.println(month + " " + map.get(month));
}
i = 0;
map = new TreeMap(new MonthNamesComparator(ru));
System.out.println("Ru Map original:");
for (String month : monthsToTestRu) {
System.out.print(month + " ");
map.put(month, new StringBuilder(String.valueOf(++i)).append(" position in source array").toString());
}
System.out.println();
System.out.println("Ru Map sorted:");
for (String month : (Set<String>)map.keySet()) {
System.out.println(month + " " + map.get(month));
}
}
}
Enjoy it, works like a charm.
Create a mapping:
month_map = {"January":1,
"February":2,
"March":3,
"April":4} # etc..
Use the mapping to compare one month to the other.
OR
Most languages/frameworks have objects for handling dates. Create date objects for all the months and compare them using the native (if available) inequality operators or basic sorting functions:
import datetime
January = datetime.date(2010,1,1)
February = datetime.date(2010,2,1)
if February < January: print("The world explodes.")
Thanks all for the suggestions, I would like to mark you all as accepted.
Here's the resulting code:
// correct order
months as String[] = ["jan", "feb", "mar", "apr", "may", "jun",
"jul", "aug", "sep", "oct", "nov", "dec"]
// my unsorted months
myMonths as String[] = ["mar", "dec", "jul", "jan", "sep"]
// poor substitute for Map
mappedIndex as Int[]
// create an array with the corresponding index
for each m in myMonths do
i as Int = 0;
for each month in months do
if m == month then
mappedIndex[] = i // no break, so I should use "else"
else
i = i + 1
end
end
end
// At this point mapped index has the same order as my "unsorted" array
// in this case [2,11,5,0,8]
// Fortunately this language has "sort" otherwise I would jump out of the window
mappedIndex.sort()
// create a new array to hold the sorted values
myMonthsSorted as String[]
// and fill it with the correct value
for each i in mappedIndex do
myMonthsSorted[] = months[i]
end
For something like months, I'd just hard-code the arrays I needed...
var correctOrdering = {
english: ["January", "February", "March", ...],
french: ["Janvier", "Février", "Mars", ...],
russian: ["Январь", "февраль", "март"],
...
};
It's not like month names are going to change any time soon.
tl;dr
EnumSet.of( Month.JANUARY , Month.MARCH , Month.OCTOBER , Month.DECEMBER ).toString()
Enum
If your language provides a powerful enum facility as does Java, define a dozen objects. See Oracle Tutorial.
java.time.Month
The java.time classes include the handy Month enum, defining a dozen objects one for each month of the year January-December.
They are numbered 1-12, and defined in proper order, January to December.
In your code base, use objects of this enum to replace any use of mere integers or use of name-of-month strings. Using Month objects throughout provides type-safety, ensures valid values, and makes your code more self-documenting.
In Java, the EnumSet and EnumMap are implementations of Set and Map that are optimized for enum values. They execute very quickly and take very little memory.
EnumSet<Month> months = EnumSet.of( Month.JANUARY , Month.MARCH , Month.OCTOBER , Month.DECEMBER );
The EnumSet iterates in natural order, the order in which the enum constants are declared. So no need to explicitly sort your collection.
The class includes a getDisplayName method for generating a localized String of the name of the month. Specify a TextStyle for how long or abbreviated you want the text. And specify a Locale for (a) the human language to use in translation, and (b) the cultural norms to decide issues such as abbreviation, punctuation, and capitalization.
for( Month month : months ) {
String output = month.getDisplayName( TextStyle.SHORT_STANDALONE , Locale.CANADA_FRENCH ); // Or Locale.US, Locale.ITALY, etc.
System.out.println( output );
}
Add a prefix for each month:
Jan -> aJan
Feb -> bFeb
...
Sort, then remove the prefix.
Related
Thanks to the great help from Tenfour04, I've got wonderful code for handling CSV files.
However, I am in trouble like followings.
How to call these functions?
How to initialize 2-dimensional array variables?
Below is the code that finally worked.
MainActivity.kt
package com.surlofia.csv_tenfour04_1
import androidx.appcompat.app.AppCompatActivity
import android.os.Bundle
import java.io.File
import java.io.IOException
import com.surlofia.csv_tenfour04_1.databinding.ActivityMainBinding
var chk_Q_Num: MutableList<Int> = mutableListOf (
0,
1, 2, 3, 4, 5,
6, 7, 8, 9, 10,
11, 12, 13, 14, 15,
16, 17, 18, 19, 20,
)
var chk_Q_State: MutableList<String> = mutableListOf (
"z",
"a", "b", "c", "d", "e",
"f", "g", "h", "i", "j"
)
class MainActivity : AppCompatActivity() {
private lateinit var binding: ActivityMainBinding
override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
// setContentView(R.layout.activity_main)
binding = ActivityMainBinding.inflate(layoutInflater)
val view = binding.root
setContentView(view)
// Load saved data at game startup. It will be invalid if performed by other activities.
val filePath = filesDir.path + "/chk_Q.csv"
val file = File(filePath)
binding.fileExists.text = isFileExists(file).toString()
if (isFileExists(file)) {
val csvIN = file.readAsCSV()
for (i in 0 .. 10) {
chk_Q_Num[i] = csvIN[i][0].toInt()
chk_Q_State[i] = csvIN[i][1]
}
}
// Game Program Run
val csvOUT = mutableListOf(
mutableListOf("0","OK"),
mutableListOf("1","OK"),
mutableListOf("2","OK"),
mutableListOf("3","Not yet"),
mutableListOf("4","Not yet"),
mutableListOf("5","Not yet"),
mutableListOf("6","Not yet"),
mutableListOf("7","Not yet"),
mutableListOf("8","Not yet"),
mutableListOf("9","Not yet"),
mutableListOf("10","Not yet")
)
var tempString = ""
for (i in 0 .. 10) {
csvOUT[i][0] = chk_Q_Num[i].toString()
csvOUT[i][1] = "OK"
tempString = tempString + csvOUT[i][0] + "-->" + csvOUT[i][1] + "\n"
}
binding.readFile.text = tempString
// and save Data
file.writeAsCSV(csvOUT)
}
// https://www.techiedelight.com/ja/check-if-a-file-exists-in-kotlin/
private fun isFileExists(file: File): Boolean {
return file.exists() && !file.isDirectory
}
#Throws(IOException::class)
fun File.readAsCSV(): List<List<String>> {
val splitLines = mutableListOf<List<String>>()
forEachLine {
splitLines += it.split(", ")
}
return splitLines
}
#Throws(IOException::class)
fun File.writeAsCSV(values: List<List<String>>) {
val csv = values.joinToString("\n") { line -> line.joinToString(", ") }
writeText(csv)
}
}
chk_Q.csv
0,0
1,OK
2,OK
3,Not yet
4,Not yet
5,Not yet
6,Not yet
7,Not yet
8,Not yet
9,Not yet
10,Not yet
1. How to call these functions?
The code below seems work well.
Did I call these funtions in right way?
Or are there better ways to achieve this?
read
if (isFileExists(file)) {
val csvIN = file.readAsCSV()
for (i in 0 .. 10) {
chk_Q_Num[i] = csvIN[i][0].toInt()
chk_Q_State[i] = csvIN[i][1]
}
}
write
file.writeAsCSV(csvOUT)
2. How to initialize 2-dimensional array variables?
val csvOUT = mutableListOf(
mutableListOf("0","OK"),
mutableListOf("1","OK"),
mutableListOf("2","OK"),
mutableListOf("3","Not yet"),
mutableListOf("4","Not yet"),
mutableListOf("5","Not yet"),
mutableListOf("6","Not yet"),
mutableListOf("7","Not yet"),
mutableListOf("8","Not yet"),
mutableListOf("9","Not yet"),
mutableListOf("10","Not yet")
)
I would like to know the clever way to use a for loop instead of writing specific values one by one.
For example, something like bellow.
val csvOUT = mutableListOf(mutableListOf())
for (i in 0 .. 10) {
csvOUT[i][0] = i
csvOUT[i][1] = "OK"
}
But this gave me the following error message:
Not enough information to infer type variable T
It would be great if you could provide an example of how to execute this for beginners.
----- Added on June 15, 2022. -----
[Question 1]
Regarding initialization, I got an error "keep stopping" when I executed the following code.
The application is forced to terminate.
Why is this?
val csvOUT: MutableList<MutableList<String>> = mutableListOf(mutableListOf())
for (i in 0 .. 10) {
csvOUT[i][0] = "$i"
csvOUT[i][1] = "OK"
}
[Error Message]
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.surlofia.csv_endzeit_01/com.surlofia.csv_endzeit_01.MainActivity}: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
In my opinion there are basically two parts to your question. First you need an understanding of the Kotlin type system including generics. Secondly you want some knowledge about approaches to the problem at hand.
type-system and generics
The function mutableListOf you're using is generic and thus needs a single type parameter T, as can be seen by definition its taken from the documentation:
fun <T> mutableListOf(): MutableList<T>
Most of the time the Kotlin compiler is quite good at type-inference, that is guessing the type used based on the context. For example, I do not need to provide a type explicitly in the following example, because the Kotlin compiler can infer the type from the usage context.
val listWithInts = mutableListOf(3, 7)
The infered type is MutableList<Int>.
However, sometimes this might not be what one desires. For example, I might want to allow null values in my list above. To achieve this, I have to tell the compiler that it should not only allow Int values to the list but also null values, widening the type from Int to Int?. I can achieve this in at least two ways.
providing a generic type parameter
val listWithNullableInts = mutableListOf<Int?>(3, 7)
defining the expected return type explicitly
val listWithNullableInts: MutableList<Int?> = mutableListOf(3, 7)
In your case the compiler does NOT have enough information to infer the type from the usage context. Thus you either have to provide it that context, e.g. by passing values of a specific type to the function or using one of the two options named above.
initialization of multidimensional arrays
There are questions and answers on creating multi-dimensional arrays in Kotlin on StackOverflow already.
One solution to your problem at hand might be the following.
val csvOUT: MutableList<MutableList<String>> = mutableListOf(mutableListOf())
for (i in 0 .. 10) {
csvOUT[i][0] = "$i"
csvOUT[i][1] = "OK"
}
You help the Kotlin compiler by defining the expected return type explicitly and then add the values as Strings to your 2D list.
If the dimensions are fixed, you might want to use fixed-size Arrays instead.
val csvArray = Array(11) { index -> arrayOf("$index", "OK") }
In both solutions you convert the Int index to a String however.
If the only information you want to store for each level is a String, you might as well use a simple List<String and use the index of each entry as the level number, e.g.:
val csvOut = List(11) { "OK" }
val levelThree = csvOut[2] // first index of List is 0
This would also work with more complicated data structures instead of Strings. You simply would have to adjust your fun File.writeAsCSV(values: List<List<String>>) to accept a different type as the values parameter.
Assume a simple data class you might end up with something along the lines of:
data class LevelState(val state: String, val timeBeaten: Instant?)
val levelState = List(11) { LevelState("OK", Instant.now()) }
fun File.writeAsCSV(values: List<LevelState>) {
val csvString = values
.mapIndexed { index, levelState -> "$index, ${levelState.state}, ${levelState.timeBeaten}" }
.joinToString("\n")
writeText(csvString)
}
If you prefer a more "classical" imperative approach, you can populate your 2-dimensional Array / List using a loop like for in.
val list: MutableList<MutableList<String>> = mutableListOf() // list is now []
for (i in 0..10) {
val innerList: MutableList<String> = mutableListOf()
innerList.add("$i")
innerList.add("OK")
innerList.add("${Instant.now()}")
list.add(innerList)
// list is after first iteration [ ["0", "OK", "2022-06-15T07:03:14.315Z"] ]
}
The syntax listName[index] = value is just syntactic sugar for the operator overload of the set operator, see the documentation on MutableList for example.
You cannot access an index, that has not been populated before, e.g. during the List's initialization or by using add; or else you're greeted with a IndexOutOfBoundsException.
If you want to use the set operator, one option is to use a pre-populated Array as such:
val array: Array<Array<String>>> = Array(11) {
Array(3) { "default" }
} // array is [ ["default, "default", "default"], ...]
array[1][2] = "myValue"
However, I wouldn't recommend this approach, as it might lead to left over, potentially invalid initial data, in case one misses to replace a value.
{
"movies": {
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson"
},
"movie2": {
"genre": "Horror",
"name": "The Shining",
"lead": "Jack Nicholson"
},
"movie3": {
"genre": "comedy",
"name": "The Mask",
"lead": "Jim Carrey"
}
}
}
I am a Firebase newbie. How can I retrieve a result from the data above where genre = 'comedy' AND lead = 'Jack Nicholson'?
What options do I have?
Using Firebase's Query API, you might be tempted to try this:
// !!! THIS WILL NOT WORK !!!
ref
.orderBy('genre')
.startAt('comedy').endAt('comedy')
.orderBy('lead') // !!! THIS LINE WILL RAISE AN ERROR !!!
.startAt('Jack Nicholson').endAt('Jack Nicholson')
.on('value', function(snapshot) {
console.log(snapshot.val());
});
But as #RobDiMarco from Firebase says in the comments:
multiple orderBy() calls will throw an error
So my code above will not work.
I know of three approaches that will work.
1. filter most on the server, do the rest on the client
What you can do is execute one orderBy().startAt()./endAt() on the server, pull down the remaining data and filter that in JavaScript code on your client.
ref
.orderBy('genre')
.equalTo('comedy')
.on('child_added', function(snapshot) {
var movie = snapshot.val();
if (movie.lead == 'Jack Nicholson') {
console.log(movie);
}
});
2. add a property that combines the values that you want to filter on
If that isn't good enough, you should consider modifying/expanding your data to allow your use-case. For example: you could stuff genre+lead into a single property that you just use for this filter.
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson",
"genre_lead": "comedy_Jack Nicholson"
}, //...
You're essentially building your own multi-column index that way and can query it with:
ref
.orderBy('genre_lead')
.equalTo('comedy_Jack Nicholson')
.on('child_added', function(snapshot) {
var movie = snapshot.val();
console.log(movie);
});
David East has written a library called QueryBase that helps with generating such properties.
You could even do relative/range queries, let's say that you want to allow querying movies by category and year. You'd use this data structure:
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson",
"genre_year": "comedy_1997"
}, //...
And then query for comedies of the 90s with:
ref
.orderBy('genre_year')
.startAt('comedy_1990')
.endAt('comedy_2000')
.on('child_added', function(snapshot) {
var movie = snapshot.val();
console.log(movie);
});
If you need to filter on more than just the year, make sure to add the other date parts in descending order, e.g. "comedy_1997-12-25". This way the lexicographical ordering that Firebase does on string values will be the same as the chronological ordering.
This combining of values in a property can work with more than two values, but you can only do a range filter on the last value in the composite property.
A very special variant of this is implemented by the GeoFire library for Firebase. This library combines the latitude and longitude of a location into a so-called Geohash, which can then be used to do realtime range queries on Firebase.
3. create a custom index programmatically
Yet another alternative is to do what we've all done before this new Query API was added: create an index in a different node:
"movies"
// the same structure you have today
"by_genre"
"comedy"
"by_lead"
"Jack Nicholson"
"movie1"
"Jim Carrey"
"movie3"
"Horror"
"by_lead"
"Jack Nicholson"
"movie2"
There are probably more approaches. For example, this answer highlights an alternative tree-shaped custom index: https://stackoverflow.com/a/34105063
If none of these options work for you, but you still want to store your data in Firebase, you can also consider using its Cloud Firestore database.
Cloud Firestore can handle multiple equality filters in a single query, but only one range filter. Under the hood it essentially uses the same query model, but it's like it auto-generates the composite properties for you. See Firestore's documentation on compound queries.
I've written a personal library that allows you to order by multiple values, with all the ordering done on the server.
Meet Querybase!
Querybase takes in a Firebase Database Reference and an array of fields you wish to index on. When you create new records it will automatically handle the generation of keys that allow for multiple querying. The caveat is that it only supports straight equivalence (no less than or greater than).
const databaseRef = firebase.database().ref().child('people');
const querybaseRef = querybase.ref(databaseRef, ['name', 'age', 'location']);
// Automatically handles composite keys
querybaseRef.push({
name: 'David',
age: 27,
location: 'SF'
});
// Find records by multiple fields
// returns a Firebase Database ref
const queriedDbRef = querybaseRef
.where({
name: 'David',
age: 27
});
// Listen for realtime updates
queriedDbRef.on('value', snap => console.log(snap));
var ref = new Firebase('https://your.firebaseio.com/');
Query query = ref.orderByChild('genre').equalTo('comedy');
query.addValueEventListener(new ValueEventListener() {
#Override
public void onDataChange(DataSnapshot dataSnapshot) {
for (DataSnapshot movieSnapshot : dataSnapshot.getChildren()) {
Movie movie = dataSnapshot.getValue(Movie.class);
if (movie.getLead().equals('Jack Nicholson')) {
console.log(movieSnapshot.getKey());
}
}
}
#Override
public void onCancelled(FirebaseError firebaseError) {
}
});
Frank's answer is good but Firestore introduced array-contains recently that makes it easier to do AND queries.
You can create a filters field to add you filters. You can add as many values as you need. For example to filter by comedy and Jack Nicholson you can add the value comedy_Jack Nicholson but if you also you want to by comedy and 2014 you can add the value comedy_2014 without creating more fields.
{
"movies": {
"movie1": {
"genre": "comedy",
"name": "As good as it gets",
"lead": "Jack Nicholson",
"year": 2014,
"filters": [
"comedy_Jack Nicholson",
"comedy_2014"
]
}
}
}
For Cloud Firestore
https://firebase.google.com/docs/firestore/query-data/queries#compound_queries
Compound queries
You can chain multiple equality operators (== or array-contains) methods to create more specific queries (logical AND). However, you must create a composite index to combine equality operators with the inequality operators, <, <=, >, and !=.
citiesRef.where('state', '==', 'CO').where('name', '==', 'Denver');
citiesRef.where('state', '==', 'CA').where('population', '<', 1000000);
You can perform range (<, <=, >, >=) or not equals (!=) comparisons only on a single field, and you can include at most one array-contains or array-contains-any clause in a compound query:
Firebase doesn't allow querying with multiple conditions.
However, I did find a way around for this:
We need to download the initial filtered data from the database and store it in an array list.
Query query = databaseReference.orderByChild("genre").equalTo("comedy");
databaseReference.addValueEventListener(new ValueEventListener() {
#Override
public void onDataChange(#NonNull DataSnapshot dataSnapshot) {
ArrayList<Movie> movies = new ArrayList<>();
for (DataSnapshot dataSnapshot1 : dataSnapshot.getChildren()) {
String lead = dataSnapshot1.child("lead").getValue(String.class);
String genre = dataSnapshot1.child("genre").getValue(String.class);
movie = new Movie(lead, genre);
movies.add(movie);
}
filterResults(movies, "Jack Nicholson");
}
}
#Override
public void onCancelled(#NonNull DatabaseError databaseError) {
}
});
Once we obtain the initial filtered data from the database, we need to do further filter in our backend.
public void filterResults(final List<Movie> list, final String genre) {
List<Movie> movies = new ArrayList<>();
movies = list.stream().filter(o -> o.getLead().equals(genre)).collect(Collectors.toList());
System.out.println(movies);
employees.forEach(movie -> System.out.println(movie.getFirstName()));
}
The data from firebase realtime database is as _InternalLinkedHashMap<dynamic, dynamic>.
You can also just convert this it to your map and query very easily.
For example, I have a chat app and I use realtime database to store the uid of the user and the bool value whether the user is online or not. As the picture below.
Now, I have a class RealtimeDatabase and a static method getAllUsersOnineStatus().
static getOnilineUsersUID() {
var dbRef = FirebaseDatabase.instance;
DatabaseReference reference = dbRef.reference().child("Online");
reference.once().then((value) {
Map<String, bool> map = Map<String, bool>.from(value.value);
List users = [];
map.forEach((key, value) {
if (value) {
users.add(key);
}
});
print(users);
});
}
It will print [NOraDTGaQSZbIEszidCujw1AEym2]
I am new to flutter If you know more please update the answer.
ref.orderByChild("lead").startAt("Jack Nicholson").endAt("Jack Nicholson").listner....
This will work.
I'm extracting AVRO data which has a JSON field that I need to get values from. The JSON has an array, and I don't know what order the different elements of the array may appear in. How can I target specific node/values?
For example, Filters[0] could be Category one time, but could be AddressType another time.
I'm extracting AVRO data - i.e.
#rs =
EXTRACT date DateTime,
Body byte[]
FROM #input_file
USING new Microsoft.Analytics.Samples.Formats.ApacheAvro.AvroExtractor(#"
...
The Body is JSON that can look like this (but Category is not always Filter[0]. This is a small example; there are 7 different types of "Field"s):
{
""TimeStamp"": ""2019-02-19T15:00:29.1067771-05:00"",
""Filters"": [{
""Operator"": ""eq"",
""Field"": ""Category"",
""Value"": ""Sale""
}, {
""Operator"": ""eq"",
""Field"": ""AddressType"",
""Value"": ""Home""
}
]
}
My U-SQL looks like this, which of course does not always work.
#keyvalues =
SELECT JsonFunctions.JsonTuple(Encoding.UTF8.GetString(Body),
"TimeStamp",
"$.Filters[?(#.Field == 'Category')].Value",
"$.Filters[?(#.Field == 'AddressType')].Value"
) AS message
FROM #rs;
#results =
SELECT
message["TimeStamp"] AS TimeStamp,
message["Filters[0].Value"] AS Category,
message["Filters[1].Value"] AS AddressType
FROM #keyvalues;
Although this does not actually answer my question, as a workaround, I modified the Microsoft 'sample' JsonFunctions.JsonTuple method to be able to specify my own key name for extracted values:
/// Added - Prefix a path expression with a specified key. Use key~$e in the expression.
/// eg:
/// JsonTuple(json, "myId~id", "myName~name") -> field names MAP{ {myId, 1 }, {myName, Ed } }
The modified code:
private static IEnumerable<KeyValuePair<string, T>> ApplyPath<T>(JToken root, string path)
{
var keySeparatorPos = path.IndexOf("~");
string key = null;
var searchPath = path;
if (keySeparatorPos > 0) // =0?if just a leading "=", i.e. no key provided, then don't parse out a key.
{
key = path.Substring(0, keySeparatorPos).Trim();
searchPath = path.Substring(keySeparatorPos + 1);
}
// Children
var children = SelectChildren<T>(root, searchPath);
foreach (var token in children)
{
// Token => T
var value = (T)JsonFunctions.ConvertToken(token, typeof(T));
// Tuple(path, value)
yield return new KeyValuePair<string, T>(key ?? token.Path, value);
}
}
For example, I can access vales and name them
#keyvalues =
SELECT JsonFunctions.JsonTuple(Encoding.UTF8.GetString(Body),
"TimeStamp",
"EventName",
"Plan~ $.UrlParams.plan",
"Category~ $.Filters[?(#.Field == 'Category')].Value",
"AddressType~ $.Filters[?(#.Field == 'AddressType')].Value"
) AS message
FROM #rs;
#results =
SELECT
message["TimeStamp"] AS TimeStamp,
message["EventName"] AS EventName,
message["Plan"] AS Plan,
message["Category"] AS Category,
message["AddressType"] AS AddressType
FROM #keyvalues;
(I've not tested to see what would happen if the same Field appears multiple times in the array; That won't happen in my case)
I would like to build a table showing the changes in stock prices over 1M, 3M, 6M, etc. The API call for a specific day in the past
returns the following json:
{
"date": "2018-01-02",
"data": {
"AAPL": {
"open": "170.16",
"close": "172.26",
"high": "172.30",
"low": "169.26",
"volume": "25555934"
},
"MSFT": {
"open": "86.13",
"close": "85.95",
"high": "86.31",
"low": "85.50",
"volume": "22483797"
}
}
}
I have built a for loop in java that goes over the required dates using Calendar and make the API calls for those dates and a list of stock symbols. I am able to deserialize the json using the following code:
JsonParser jsonParser = new JsonParser();
JsonObject jsonObject = (JsonObject) jsonParser.parse(result);
JsonElement jsonElement = jsonObject.get("data");
Set<Map.Entry<String, JsonElement>> entrySet = jsonElement.getAsJsonObject().entrySet();
entrySet.parallelStream().forEach(entry -> {
Stock stk = new Stock();
stk.setSymbol(entry.getKey());
stk.setClose(entry.getValue().getAsJsonObject().get("close").getAsFloat());
stk.setDate(date.getKey());
The problem is that using this code I can only save the data in my database that is unique by date. I wish to save it so that is unique by stock symbol. Presumably I need to collect the dates in a map object as a property to the POJO. Unfortunately, I have not been able to make it work. Any suggestions would be very much appreciated.
Create a flat table in your database, with columns for the date, the symbol (such as AAPL), and the five numbers. So seven columns all together.
This strategy assumes you have no additional data to store per date or per stock. (If you did have additional data per date or stock, you would need multiple tables, with parent tables for the date or the stock.)
Personally, I would slap on another column with a UUID generated as a primary key. I believe in always using surrogate key. But not necessary strictly speaking, if your database supports a multi-column combination as a natural key. You could combine the date and the symbol together as a primary key to uniquely identify each row.
CREATE TABLE daily_high_low_ (
date_of_closing_ DATE ,
symbol_ VARCHAR( 4 ) ,
open_ NUMERIC( 12 , 2 ) ,
close_ NUMERIC( 12 , 2 ) ,
high_ NUMERIC( 12 , 2 ) ,
low_ NUMERIC( 12 , 2 ) ,
volume_ INTEGER ,
PRIMARY KEY ( date_of_closing_ , symbol_ )
)
I am purposely keeping this overly simplistic. In real life you would have to worry about stock ticker symbols not being unique across stock exchanges. And I have hard-coded the numeric precision and scale for US dollars and cents amounts, which may not be appropriate for all markets. And we are ignoring the fact that date varies across time zones around the globe.
As you process your JSON, flatten the data to fit this flat table.
For retrieval, make a class to match this database table.
public class HighLow {
public LocalDate dateOfClosing ;
public String symbol ;
public BigDecimal open, close, high, low ;
public Integer volume ;
}
Load from database into a data structure that meets the needs of your data analysis. Perhaps hard-coded maps for each time period: months1, months3, months6, etc. Each maps the symbol to a list of HighLow objects.
To save memory, you share the same objects across the lists. For example, the list in the months3 map has the same HighLow objects from the list in the months1 map, plus two more months worth of HighLow objects.
Map< String , List< HighLow > > months1 = new TreeMap<>() ;
Map< String , List< HighLow > > months3 = new TreeMap<>() ;
Map< String , List< HighLow > > months6 = new TreeMap<>() ;
I''m using EF to query the database using anonymous type.
here the code I use for EF
public JsonResult OverdueEventsCustom()
{
var eventCustomOverdue = _eventCustomRepository.FindOverdueEventsCustom();
return Json(eventCustomOverdue, JsonRequestBehavior.AllowGet);
}
public IQueryable<dynamic> FindOverdueEventsCustom()
{
DateTime dateTimeNow = DateTime.UtcNow;
DateTime dateTomorrow = dateTimeNow.Date.AddDays(1);
return db.EventCustoms.Where(x => x.DateTimeStart < dateTomorrow)
.Select(y => new { y.EventId, y.EventTitle, y.DateTimeStart});
}
Inspecting using the debugger I see the properties is in this format
Date = {16/08/2012 00:00:00}
The resultfor the JSON is
[{
"EventId": 1,
"EventTitle": "Homework Math",
"DateTimeStart": "\/Date(1345108269310)\/"
}, {
"EventId": 4,
"EventTitle": "Homework help with Annie",
"DateTimeStart": "\/Date(1345108269310)\/"
}, {
"EventId": 6,
"EventTitle": "Physic laboratory",
"DateTimeStart": "\/Date(1345108269310)\/"
}]
I need the the json in this format
"DateTimeStart": "(16/08/2012)"
Any idea what i'm doing wrong here? thanks for your help
Related articles
http://www.hanselman.com/blog/OnTheNightmareThatIsJSONDatesPlusJSONNETAndASPNETWebAPI.aspx
How do I format a Microsoft JSON date?
"\/Date(1345108269310)\/" is the correct way to pass a Date to javascript. The way I see it, you have two options here:
If you do not explicitly need the value as a date, you could just pass a string to the JSON variable, containing the pretty-printed date.
Something along the lines of:
DateTimeStart: String.Format("{0: dd-MM-yyyy}", myDate)
If you will still need to use the variable a a date in javascript (for calculations for example), the most consice and readably way would be to create a javascript function that converts said date into the pretty-printed string you want (I don't know if such a function already exists. It isn't too hard to create though:
function prettyDate(date) {
return date.getDate() + "-" + date.getMonth() + "-" + date.getFullYear();
}
I would suggest passing it along as a string from you code behind, as it is more readable. But that only works if you do not need to use the date except for displaying.