I have the following code for converting datetime in a json file to the desired format (%Y-%m-%d), but it gave me the error code of " list indices must be integers or slices, not str", highlighting the tweet["created_at"] part.
counts = {}
for tweet in tweet_list:
date = datetime.strftime(datetime.strptime(tweet["created_at"],'%a %b %d %H:%M:%S +0000 %Y'), '%Y-%m-%d')
if date not in counts:
counts[date] = dict(sentiments=list(), tweet_count = 0)
counts[date]["tweet_count"] += 1
full_text = cleanTxt(tweet["text"])
counts[date]["sentiments"].append(find_sentiment(text))
The tweet_list looks like this, I suspect when I converted the dataframe into json using orient='records', it kind of messed up the formatting?
To work around this, I tired using orient = 'index' but the final json file will include the row number and can't get ride of it.
Related
I want to get this output:
2018-09-02 00:00:00.000
I tried the below code:
.Cells(LRS + 1, 15).Value = Format(.Cells(LRS + 1, "A").Value, "yyyy-MM-dd hh:mm:ss.fff")
And I got:
2018-09-02 00:00:00.fff
The initial date in Excel has the following format yyyy/mm/dd, no time included. That's why the time part includes only zeros 00:00:00.000. The reason I want to include the time in the specific format is that I'm planning to import those dates into a SQL table with that format.
Is there any solution?
As you can see from the documentation fff is not recognised as a formatting token in VBA.
Helpfully, you can actually import your data into SQL without formatting it to add the time. If you import it into a datetime field the SQL engine will automatically default the time part of the field to midnight on the date you give it.
I think you can just change your format string to yyyy-MM-dd by itself.
However if you really want to do it like this, then since there's no time specified then just hard-code 000 instead of fff. The rest of the time can be similarly hard-coded, since it never varies, so you end up with yyyy-MM-dd 00:00:00.000. But as I said, I think it's a bit pointless.
After replacing the cell format with the corresponding format, it is likely that the value of the cell is imported as text, not as a value.
Sub test()
Dim s As String, s1 As String, s2 As String
'First Cell format as your "yyyy-mm-dd hh:mm:ss.000"
Range("a2").NumberFormatLocal = "yyyy-mm-dd hh:mm:ss.000"
'In vb,This format("yyyy-mm-dd hh:mm:ss.000") is not recognized.
Range("b2") = Format(Range("a2"), "yyyy-mm-dd hh:mm:ss.000")
s1 = Format(Range("a2"), "yyyy-mm-dd hh:mm:ss.000")
's1 = "2018-09-03 01:24:33.000"
'Since you format a2 cell as "yyyy-mm-dd hh:mm:ss.000" you can get data as text
Range("b2") = Range("a2").Text
s2 = Range("a2").Text
's2= "2018-09-03 01:24:33.240"
End Sub
Sheet Data
Local Window
Using Python 2.7, I wont to pass a query from BigQuery to ML Predict which has a specific formating request.
First: Is there an easier way to go directly from the BigQuery query to JSON in the correct format so it can be passed to requests.post() instead of going through pandas (from what I understand pandas is still not supported for GCP Standard)?
Second: Is there a way to construct the query to go directly to a JSON format and then modify the JSON to reflect the ML Predict JSON requirments?
Currently my code looks like this:
#I used the bigquery to dataframe option here to view the output.
#I would like to not use pandas in the end code.
logs = log_data.execute(output_options=bq.QueryOutput.dataframe()).result()
data = logs.to_json(orient='index')
print data
'{"0":{"end_time":"2018-04-19","device":"iPad","device_os":"iOS","device_os_version":"5.1.1","latency":0.150959,"megacycles":140.0,"cost":"1.3075e-08","device_brand":"Apple","device_family":"iPad","browser_version":"5.1","app":"567","ua_parse":"0"}}'
#The JSON needs to be in this format according to google documentation.
#data = {
# 'instances': [
# {
# 'key':'',
# 'end_time': '2018-04-19',
# 'device': 'iPad',
# 'device_os': 'iOS',
# 'device_os_version': '5.1.1',
# 'latency': 0.150959,
# 'megacycles':140.0,
# 'cost':'1.3075e-08',
# 'device_brand':'Apple',
# 'device_family':'iPad',
# 'browser_version':'5.1',
# 'app':'567',
# 'ua_parse':'40.9.8'
# }
# ]
#}
So all I would need to change is the leading key '0' to 'instances' and I should be all set to pass into `requests.post().
Is there a way to accomplish this?
Edit-Adding BigQuery query:
%%bq query --n log_data
WITH `my.table` AS (
SELECT ARRAY<STRUCT<end_time STRING, device STRING, device_os STRING, device_os_version STRING, latency FLOAT64, megacycles FLOAT64,
cost STRING, device_brand STRING, device_family STRING, browser_version STRING, app STRING, ua_parse STRING>>[] instances
)
SELECT TO_JSON_STRING(t)
FROM `my.table` AS t
WHERE end_time >='2018-04-19'
LIMIT 1
data = log_data.execute().result()
Thanks to #MikhailBerlyant I have adjust my query and code to look like this:
%%bq query --n log_data
SELECT [TO_JSON_STRING(t)] AS instance
FROM `yourproject.yourdataset.yourtable` AS t
WHERE end_time >='2018-04-19'
LIMIT 1
But when I run the execute logs = log_data.execute().result() I get this
Which results in this error when passing into request.post
TypeError: QueryResultsTable job_zfVEiPdf2W6msBlT6bBLgMusF49E is not JSON serializable
Is there a way within execut() to just return the json?
First: Is there an easier way to go directly from the BigQuery query to JSON in the correct format
See example below
#standardSQL
WITH yourTable AS (
SELECT ARRAY<STRUCT<id INT64, type STRING>>[(1, 'abc'), (2, 'xyz')] instances
)
SELECT TO_JSON_STRING(t)
FROM yourTable t
with result is in the format you asked for:
{"instances":[{"id":1,"type":"abc"},{"id":2,"type":"xyz"}]}
Above demonstrates the query and how it will work
In you real case - you should use something like below
SELECT TO_JSON_STRING(t)
FROM `yourproject.yourdataset.yourtable` AS t
WHERE end_time >='2018-04-19'
LIMIT 1
hope this helps :o)
Update based on comments
SELECT [TO_JSON_STRING(t)] AS instance
FROM `yourproject.yourdataset.yourtable` t
WHERE end_time >='2018-04-19'
LIMIT 1
I wanted to add this in case someone has the same problem I had or at least are stuck on were to go once you have the query.
I was able to write a function that formatted the query in the way Google ML Predict wants it to be passed into requests.post(). This is most likely a horrible way to accomplish this but I could not find a direct way to go from BigQuery to ML Predict in the correct format.
def logs(query):
client = gcb.Client()
query_job = client.query(query)
CSV_COLUMNS ='end_time,device,device_os,device_os_version,latency,megacycles,cost,device_brand,device_family,browser_version,app,ua_parse'.split(',')
for row in query_job.result():
var = list(row)
l1 = dict(zip(CSV_COLUMNS,var))
l1.update({'key':''})
l2 = {'instances':[l1]}
return l2
I can't figure out how to tell VBA the correct date format, as it seems to assume in the wrong way around. So CDate("24/02/2016 10:59") is giving the correct date - dd/mm/yyy. However when it iterates through a date like CDate("01/03/2016 00:59") - it assumes number 03 is not a month but a day so VBA is assuming it's mm/dd/yyyy, so it's giving me a date 03/01/2016 which is wrong and messing my data. So how can I tell VBA it's actually dd/mm/yyyy format. VBA seems to automatically pick up nicely even if it's "2016/01/14", but it's failing when it's not really obvious which part of numbers are months e.g. both dd and mmare less than 12.
I'm reading this date from a CSV file using WS.Cells(irow, icol).Value.
This is what I tried so far:
datDate = CDate(Format(WS.Cells(iRow, iCell).Value, "dd-mmm-yyyy hh:mm:ss"))
When the CDate() function encounters an ambiguous "aa/bb/yyyy" date string it will be interpreted according to the order that the Month and Day appear in the short date format defined by the Regional Settings in Windows.
For example:
When my machine is set to "English (United States)" with a short date format of M/d/yyyy then Month(CDate("02/04/2016")) returns 2.
However, after changing my Regional Settings to "English (Canada)" with a short date format of dd/MM/yyyy then Month(CDate("02/04/2016")) returns 4.
Note that this is different from the behaviour of the Jet/ACE database engine when interpreting ambiguous #aa/bb/yyyy# date literals in SQL statements. In that case it will always interpret them as mm/dd/yyyy regardless of the regional settings.
The best solution, as always, is to ensure that the string representation uses an UNambiguous date format, e.g., 2016/02/04 will always be interpreted as Feb 4.
Use mm/dd/yyyy hh:nn:ss like:
#03/01/2016 16:32:58#
or DateSerial and TimeSerial:
DateSerial(2016, 03, 01) + TimeSerial(16, 32, 58)
To parse the string, use Mid:
TrueDate = Dateserial(Mid(s, 7, 4), Mid(s, 4, 2), Mid(s, 1, 2)) + TimeSerial(Mid(s, 12, 2), Mid(s, 15, 2), 0)
s = "01/03/2016 00:59"
' Returns: 2016-03-01 00:59:00
s = "24/02/2016 10:59"
' Returns: 2016-02-24 10:59:00
Change the format of the date before you convert. Something like this;
Dim strDate As String, strFormatDate As String
Dim dte As Date
strDate = "01/03/2016 00:59"
strFormatDate = Format(strDate, "dd-mmm-yyyy hh:nn")
dte = CDate(strFormatDate)
This changes the date string to 01-Mar-2016 00:59 which should convert to a date data type without confusion.
Python 3.4 ... I am trying to convert a specific column in CSV from string to datetime and write it to the file. I am able to read and convert it, but need help writing the date format change to the file (the last line of code - needs to be modified). I got the following error for last line of code Error: sequence expected.
import csv
import datetime
with open ('/path/test.date.conversion.csv') as csvfile:
readablefile = csv.reader(csvfile)
writablefile = csv.writer(csvfile)
for row in readablefile:
date_format = datetime.datetime.strptime(row[13], '%m/%d/%Y')
writablefile.writerow(date_format)
So date_format is an instance of the datetime class. You need to format a string to print out to file. You can go with the standard:
date_format.isoformat()
Or customize your own format:
date_format.strftime('%Y-%m-%d %H:%M:%S')
Also a couple more things to note. You can't write to the same file you're reading from like that. If you want to overwrite that file, create a tmp file then replace it when done, but it's better to just create another file. The best solution imho is to make good use of stdin and stdout like this:
import csv
import sys
from datetime import datetime
FIELD = 13
FMT = '%m/%d/%Y'
readablefile = csv.reader(sys.stdin)
writablefile = csv.writer(sys.stdout)
for row in readablefile:
dt = datetime.strptime(row[FIELD], FMT)
row[FIELD] = dt.isoformat()
writablefile.writerow(row)
Now you can call the script like so:
cat /path/test.date.conversion.csv | python fixdate.py > another_file.csv
At this point, you can make use of the argparse module to provide all kinds of nice parameters to your script like:
Which field to format?
What input and/or output format?
etc.
I was trying to convert this format of String to Date and was unsuccessfull
"23-DEC-2008" to a Date Object ,it looks like its not accepting "-" and i could see NULL in the date object after formatting .
Can somebody let me know if u have come across this problem .
Thanks ,
Sudeep
The "-" shouldn't be an issue. I've converted SQL timestamp strings to Date objects with no problem (the format of SQL date strings is YYYY-MM-DD). What is the format string you're using? Try using the format string "DD-MMM-YYYY" and see if that works.
Edit
Sorry, my solution only applies to the DateFormatter class from Flex, and not Actionscript. After looking at the documentation for the Actionscript Date class I saw the following:
The year month and day terms can be separated by a forward slash (/) or by spaces, but never by a dash (-). (1)
If you're stuck using straight Actionscript, it looks as if you'll have to write your own parse method that accepts the "-".
this sort of works ...
public function parse(source:String):Date {
var ret:Date = new Date(0, 0, 0, 0, 0, 0, 0);
var parts:Array = source.split("-");
ret.fullYear = Number(parts[2]);
ret.setDate(Number(parts[0]));
var month:int = "jan,feb,mar,apr,may,jun,jul,aug,sep,oct,now,dec".split(",").indexOf(String(parts[1]).toLowerCase());
if (month == -1) throw "could not parse month";
ret.setMonth(month);
return ret;
}
but really, i don't like it ... if 'DEC' were 'Dec' , then
Date.parse("23-Dec-2008".split("-").join(" "))
would work ... but still ... i think, you should get something more robust ...
greetz
back2dos