Date parsing and when to use utc/TimeUnits in Vega Lite? - vega-lite

I am attempting to understand how date parsing in Vegalite works. Specifically, I am a bit confused in my understanding of default timezone assumptions and date parsing from a non-timezone denoted string.
Consider the minimal example that works
{
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
"data": {
"values": [
{"date": "2020-10-01", "distance": 1},
{"date": "2020-11-01", "distance": 5}
]
},
"mark": "bar",
"encoding": {
"x": {"field": "date",
"type": "temporal",
"timeUnit": {"unit": "yearmonthdate", "utc": true},
"axis": {"format": "%b. %y"}
},
"y": {"field": "distance", "aggregate": "sum"}
}
}
In the above example, if I omit the line (or just the utc flag):
"timeUnit": {"unit": "yearmonthdate", "utc": true}
the dates seem to get parsed as:
Wed, 30 Sep 2020 05:00:00 GMT
Sat, 31 Oct 2020 05:00:00 GMT
Any guidance or explanation on the default assumption here would be extremely helpful. I understand from the docs that given non-ISO string inputs, Vega will parse times as local (https://vega.github.io/vega-lite/docs/timeunit.html#utc) but that does not seem to be the case here?
Thank you

There are two important things to know about how Vega/Vega-Lite handles dates:
Dates are always displayed in local time, unless otherwise specified (e.g. by passing "utc": true to a timeUnit)
Dates are parsed using standard javascript date parsing.
Why is #2 important? Well, because Javascript date parsing assumes different timezones depending on how input dates are formatted, and the timezones used can even depend on what browser you are using! (read more than you ever wanted to know at https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse#Description)
The distilled summary is that full ISO date strings will (on all browsers) be parsed as local time (I ran this on a computer set to PDT):
> new Date("2020-10-01T00:00:00")
Thu Oct 01 2020 00:00:00 GMT-0700 (Pacific Daylight Time)
whereas partial dates or timestamps will (on most browsers) be parsed as UTC time:
> new Date("2020-10-01")
Wed Sep 30 2020 17:00:00 GMT-0700 (Pacific Daylight Time)
What this means is that if you are passing non-ISO time strings to Vega-Lite, you must use UTC time units on the axes in order to see the correct representation of the data in the vega/vega-lite chart. If you do not, the dates will be parsed in UTC time and displayed in local time, resulting in an offset equal to the timezone offset of the browser used to view the visualization.

Related

update unix timestamps (in ms) with their human-readable equivalents in a JSON value

I have a file with some epoch date (with millisecondes) coded inside.
I would like to parse the file to convert the epoch into human readable date format.
This is what the content of the file looks like :
{
"Date": "1555095683495",
"read": "1"
},
{
"Date": "1555578329532",
"read": "1"
},
I tried to convert the date by using the commande date with the use into sed but dont get it working correctly.
=> case 1 : the epoch date is replaced but date is not executed.
$ cat myfile.json | sed -e "s/\([0-9]\{13,\}\)/date -d #\1/g"*
{
"Date": "date -d #1555095683495",
"read": "1"
},
{
"Date": "date -d #1555578329532",
"read": "1"
}
==> case 2: epoch date is replace by a executed command of date but it take #1 as value and not the good one...
$cat myfile.txt | sed -e "s/[0-9]\{13,\}/$(date -d #\1)/g"
{
"Date": "jeu 01 jan 1970 01:00:01 CET",
"read": "1"
},
{
"Date": "jeu 01 jan 1970 01:00:01 CET",
"read": "1"
}
Any help appreciated to get it working correctly.
I also presume the value has to be divided by 1000 to get the milliseconds away but dont see how to get both working.
With JQ, convert each timestamp to a number, divide the result by 1000 to get a value that date manipulation builtins could understand, and feed it to strflocaltime to get a human-readable date string.
$ jq '.[].Date |= (tonumber / 1000 | strflocaltime("%c %Z"))' file
[
{
"Date": "Fri Apr 12 22:01:23 2019 +03",
"read": "1"
},
{
"Date": "Thu Apr 18 12:05:29 2019 +03",
"read": "1"
}
]

How to parse json lists in flutter

How can we parse such type of List data in Flutter? I tried different models but failed. Following is the data:
[
{
"course":"Computer Architecture Fall 2018 - KHI",
"pending_assignment":[
{
"name":"OOAD Project Fall 2018 Section B & D",
"dueDate":"Dec 14, 2018 10:00 pm"
}
]
},
{
"course":"Object Oriented Analysis and Design Fall 2018 - KHI",
"pending_assignment":[
{
"name":"OOAD Project Fall 2018 Section B & D",
"dueDate":"Dec 14, 2018 10:00 pm"
}
]
}
]
As you can check with this website (https://jsonformatter.org/json-parser), that JSON is not valid and contains errors.
For how to parse JSON in general, you can use the json.decode(jsonString); function from the dart:convert package. For more information on that you can check out this site: https://flutter.io/docs/development/data-and-backend/json

Need help in parsing the below using reex

I am zero in regex and need help in parsing the value for key "access_token" from the below output.This will be later used in passing as variable for another function.
So basically the regex should only fetch
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJodHRwOi8vc2NoZW1hcy54bWxzb2FwLm9yZy93cy8yMDA1LzA1L2lkZW50aXR5L2NsYWltcy9uYW1lIjoiaXRwcm9kbW9uaXRvciIsImh0dHA6Ly9zY2hlbWFzLnhtbHNvYXAub3JnL3dzLzIwMDUvMDUvaWRlbnRpdHkvY2xhaW1zL2dpdmVubmFtZSI6IkxPS0VTSCBEVVJBSVJBSiIsImh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vd3MvMjAwOC8wNi9pZGVudGl0eS9jbGFpbXMvcm9sZSI6IlNEQ1NfSEVMUERFU0siLCJQZXJtaXNzaW9ucyI6IjkwMzAsOTAwMCw5MDI4LDkwMjcsOTAyNiw5MDI1LDkwMjQsOTAyMyw5MDIyLDkwMjEsOTAyMCw5MDE5LDkwMTgsOTAxMyw5MDEyLDkwMTEsOTAxMCw5MDA5LDkwMDgsOTAwNyw5MDA2LDkwMDUsOTAwNCw5MDAzLDkwMDIsOTAwMSw5MDI5IiwiYWlycG9ydHMiOiJTWVoiLCJjbGllbnRJUCI6IjEwLjExMS4xLjEiLCJlbnYiOiJQUk9EIiwicmVzQ2hhbm5lbElEIjoiMTkiLCJpc0FwdENsbnQiOiJUcnVlIiwiY2hrTGNuIjoiQWlycG9ydCIsIm5iZiI6MTUzODkxNDY0OSwiZXhwIjoxNTM4OTE3NjQ5LCJpc3MiOiJmbHlkdWJhaS5jb20iLCJhdWQiOiIxNDEyMDAxIn0.qYID1b5lMjFhn7fTcSX5v6K6z2YpGJwAvE4gQfVrhxo
Here is the output of my Post output
{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJodHRwOi8vc2NoZW1hcy54bWxzb2FwLm9yZy93cy8yMDA1LzA1L2lkZW50aXR5L2NsYWltcy9uYW1lIjoiaXRwcm9kbW9uaXRvciIsImh0dHA6Ly9zY2hlbWFzLnhtbHNvYXAub3JnL3dzLzIwMDUvMDUvaWRlbnRpdHkvY2xhaW1zL2dpdmVubmFtZSI6IkxPS0VTSCBEVVJBSVJBSiIsImh0dHA6Ly9zY2hlbWFzLm1pY3Jvc29mdC5jb20vd3MvMjAwOC8wNi9pZGVudGl0eS9jbGFpbXMvcm9sZSI6IlNEQ1NfSEVMUERFU0siLCJQZXJtaXNzaW9ucyI6IjkwMzAsOTAwMCw5MDI4LDkwMjcsOTAyNiw5MDI1LDkwMjQsOTAyMyw5MDIyLDkwMjEsOTAyMCw5MDE5LDkwMTgsOTAxMyw5MDEyLDkwMTEsOTAxMCw5MDA5LDkwMDgsOTAwNyw5MDA2LDkwMDUsOTAwNCw5MDAzLDkwMDIsOTAwMSw5MDI5IiwiYWlycG9ydHMiOiJTWVoiLCJjbGllbnRJUCI6IjEwLjExMS4xLjEiLCJlbnYiOiJQUk9EIiwicmVzQ2hhbm5lbElEIjoiMTkiLCJpc0FwdENsbnQiOiJUcnVlIiwiY2hrTGNuIjoiQWlycG9ydCIsIm5iZiI6MTUzODkxNDY0OSwiZXhwIjoxNTM4OTE3NjQ5LCJpc3MiOiJmbHlkdWJhaS5jb20iLCJhdWQiOiIxNDEyMDAxIn0.qYID1b5lMjFhn7fTcSX5v6K6z2YpGJwAvE4gQfVrhxo",
"token_type": "bearer",
"expires_in": 2999,
"refresh_token": "65be41084c0b4adeaeec9725cb2e6240",
"audience": "1412001",
"displayName": "Lokesh",
"userId": "testuser78",
"rolesandpermission": "HELPDESK:9030,9000,9028,9027,9026,9025,9024,9023,9022,9021,9020,9019,9018,9013,9012,9011,9010,9009,9008,9007,9006,9005,9004,9003,9002,9001,9029",
"resChannelID": "19",
"Client": "FYC",
"isAptClnt": "True",
"scope": "apt:FYC env:PROD role:HELPDESK",
".issued": "Sun, 07 Oct 2018 12:17:29 GMT",
".expires": "Sun, 07 Oct 2018 13:07:29 GMT"
}
Simple PCRE look like
\"access_token\":.\"(.+?)\"
And you will get your token in fast captured group
You can practice with this regex on this website
https://regex101.com/

How can I store a time value, specifically only hour and minute in Firebase?

I don't need this time associated with any date. In fact, I don't want it associated with a specific date. It's more so associated with a generic day of the week. Monday, Saturday, etc.
For example, your local coffee shop sets their hours for Monday based on the fact that it is a Monday. Not based on it being May 22, 2017 or May 29, 2017, etc.
I could do something like...
"Local Coffee Shoppe":
{
"availability":
{
"monday":
{
"start":
{
"hour": 7,
"minute": 30
},
"end":
{
"hour": 21,
"minute": 0
}
}
}
}
This would mean the Local Coffee Shoppe is open Mondays from 7:30am to 9:00pm.
Is this the best way to achieve this? All times would be stored in UTC for example, with conversion happening client side.
EDIT1: Just realized there is a hole in this approach. It doesn't cover establishments that are open past midnight. To combat this, maybe each day just stores "start" and then "duration"...
"monday":
{
"start":
{
"hour": 7,
"minute": 30
},
"duration":
{
"hour": 13,
"minute": 30
}
}
So it is open for 13 hours and 30 minutes past 7:30am. I.E. 9:00pm.
I would recommend just storing timestamps and extracting as you suggested, that would be a good approach, this would also allow you to already have the code to expand your use case if you want.

how to convert the Elasticsearch json output in a table?

I'm using Head plugin for ELASTICSEARCH for running queries.
I want to convert in a table the output of the query.
The part that I need is just the "hits" object array
where the columns are the fields that I have specified into the query:
"http.date","src_shift","#timestamp","src_tz".
is there any tool or plugin to do that?
below a brief output of query:
"took": 2418,
"timed_out": false,
"_shards": {
"total": 3503,
"successful": 3503,
"failed": 0
},
"hits": {
"total": 2524,"max_score": 9.194927,"hits": [
{
"_index": "$002555","_type": "pcap","_id": "AVAJJphp2MeWtoWCbQYG","_score": 9.194927,"fields": {
"src_shift": [
1],"http.date": [
"Fri, 12 Jun 2015 22:40:54 GMT"],"#timestamp": [
1434147980397],"src_tz": [
"Europe/Warsaw"]}},{
"_index": "$002555","_type": "pcap","_id": "AVAJJphp2MeWtoWCbQYH","_score": 9.194927,"fields": {
"src_shift": [
1],"http.date": [
"Fri, 12 Jun 2015 22:40:54 GMT"],"#timestamp": [
1434147980397],"src_tz": [
"Europe/Warsaw"]}},...
In the head plugin, on your Any Request tab, you can use the Result Transformer section located just below the Query section. By default it returns the whole JSON response.
You can modify that and massage the response to return whatever you want. In your case, if you replace the default return root; by the code below, you'll get what you want:
return root.hits.hits.map(function(hit) {
var values = [];
for (var field in hit.fields) {
values.push(hit.fields[field]);
}
return values.join(",");
});
The output should be
1,"Fri, 12 Jun 2015 22:40:54 GMT",1434147980397,"Europe/Warsaw"
1,"Fri, 12 Jun 2015 22:40:54 GMT",1434147980397,"Europe/Warsaw"
...
There's a utility in Kibana called tabify that converts ElasticSearch results into tabular form. You can find its implementation here: https://github.com/elastic/kibana/blob/master/src/ui/public/agg_response/tabify/tabify.js