I'm using MySQL - Rows to Columns and this tutorial http://stratosprovatopoulos.com/web-development/mysql/pivot-a-table-in-mysql/#comment-6128 to help me pivot a table and it's working pretty well. Starting with this:
mediaID q_short_name start_time stop_time audio_link
ee CVV Number 208 210 j.mp3
ee Expiration Date 308 310 j.mp3
ff CVV Number 124 127 k.mp3
ff Expiration Date 166 169 k.mp3
The goal is this:
mediaID CVVNumStartT CVVNumStopT ExpDateStart_time ExpDateStop_time Aud
ee 208 210 308 310 k.mp3
ff 124 127 166 169 j.mp3
I got part of the way there with this code:
CREATE VIEW test__extension AS (
SELECT amr_text.*,
CASE WHEN q_short_name = 'CVV Number' THEN amr_text.start_time END AS
CVV_Start_Time,
CASE WHEN q_short_name = 'CVV Number' THEN amr_text.stop_time END AS
CVV_Stop_Time,
CASE WHEN q_short_name = 'Expiration Date' THEN amr_text.start_time END
AS Expiration_Start_Time,
CASE WHEN q_short_name = 'Expiration Date' THEN amr_text.stop_time END
AS Expiration_Stop_Time, FROM amr_text);
CREATE VIEW test_extension_pivot AS (SELECT mediaID,
SUM(CVV_Start_Time) AS CVV_Start_Time,
SUM(CVV_Stop_Time) AS CVV_Stop_Time,
SUM(Expiration_Start_Time) AS Expiration_Start_Time,
SUM(Expiration_Stop_Time) AS Expiration_Stop_Time,
FROM test_extension GROUP BY mediaID);
This creates columns exactly like the goal table. But now the values for everything except the mediaIDs are rendered as NULL. My questions are, why did they get replaced by NULL, and what can I use instead of SUM to render the values of Expiration and CVV Start and Stop Time as they are in the original table?
Currently I am using GO-GORM for all of my database queries (mostly CRUD) and I am having some issues inserting a generated UUID into a MySQL database column.
The column is a BINARY(16) as suggested in multiple blogs, the UUID is generated using github.com/satori/go.uuid package for Golang.
I am using GORM's BeforeCreate hook to generate the UUID if one does not already exist on the user, the code that I am using is as follows:
func (u *User) BeforeCreate(scope *gorm.Scope) (err error) {
if u.UserID == uuid.Nil {
uuid, err := uuid.NewV4().MarshalBinary()
scope.SetColumn("user_id", uuid)
}
}
I have also used len to get the length that MarshalBinary outputs and it returns as 16.
The error I get from GORM when trying to insert the UUID into MySQL is as follows:
(Error 1406: Data too long for column 'user_id' at row 1)
I have also fmt.Println(uuid) to see the results and they are also as follows (obviosuly changes as the UUID is generated every insert)
[93 132 59 55 102 96 72 35 137 185 34 21 195 88 213 127]
My MYSQL schema is as follows also:
CREATE TABLE users
(
id INT(10) unsigned PRIMARY KEY NOT NULL AUTO_INCREMENT,
created_at TIMESTAMP,
updated_at TIMESTAMP,
deleted_at TIMESTAMP,
user_id BINARY(16) NOT NULL,
username VARCHAR(255) NOT NULL,
password VARCHAR(255),
firstname VARCHAR(255),
lastname VARCHAR(255),
email VARCHAR(255),
address_id VARCHAR(255)
);
CREATE INDEX idx_users_deleted_at ON users (deleted_at);
CREATE UNIQUE INDEX username ON users (username);
CREATE UNIQUE INDEX user_id ON users (user_id);
I have tried different methods and libraries to generate UUIDs and convert them to binary to insert with similar results.
I think the problem is in the definition of model User. To save the GUID as 16-bytes binary, you need to define the UserID column as []byte not uuid.UUID.
type User struct {
//other fields ..
UserID []byte
//other fields ...
}
func (u *User) BeforeCreate(scope *gorm.Scope) (err error) {
if u.UserID == nil {
uuid, err := uuid.NewV4().MarshalBinary()
scope.SetColumn("user_id", uuid)
}
return nil
}
If you define the field as uuid.UUID, gorm "misinterpreted" the field as string and then insert that string into the database as binary. For example, the following UUID,
uuid: 16ac369b-e57f-471b-96f6-1068ead0bf98
//16-bytes equivalent
bytes: [22 172 54 155 229 127 71 27 150 246 16 104 234 208 191 152]
will be inserted to database as the ASCII codes of the UUID which are
0x31 0x36 0x61 0x63 0x33 0x36 0x39 0x62 0x2D 0x65 ...
('1' '6' 'a' 'c' '3' '6' '9' 'b' '-' 'e' ...)
which are 36-bytes in length, thus you're getting Error 1406: ...
In SSRS report I have a field with percentage values from which I have calculated cumulative running total. On the third field user wants to see only values which are closest to every ten value and blank out everything else.
So in the example we show 8 as cumulative value, as it is closest to 10. For the second value we choose 20 as it is closest value to 20. For the third we take 32, closest to 30. then 40, 52 , 62, 73, 79, 91
% cumulative val showed values
3 3
5 8 8
6 14
3 17
2 19
1 20 20
4 24
3 27
5 32 32
7 39
1 40 40
2 42
2 44
3 47
5 52 52
2 54
3 57
1 58
4 62 62
3 65
1 66
7 73 73
2 75
1 76
3 79 79
4 83
2 85
3 88
1 89
2 91 91
I have tried to use the solution with different set of records and that's what I can see
Result Set
Whenever I find questions like this in reporting-services tag I think about the possibilities users could have if there were something like a NEXT() function, contrary to the PREVIOUS() supported function.
Okay, I vented.
I recreated the table you posted in the question and added row number to the dataset.
Go to Report menu / Report Properties / Code tab, in the text area put the following code.
Dim running_current as Integer = 0
Dim flag as Integer = 0
Public Function Visible(ByVal a as Integer,ByVal b as Integer) As String
running_current = running_current + a
Dim running_next as Integer = running_current + b
Dim a_f as Integer = CInt(Math.ceiling(running_current / 10.0)) * 10
Dim b_f as Integer = CInt(Math.ceiling(running_next / 10.0)) * 10
if flag = 1 Then
flag = 0
return "true"
End If
If a_f = b_f Then
return "false"
Else
IF Closest(running_current,running_next) = "false" Then
flag = 1
return "false"
End If
return "true"
End if
End Function
Public Function Closest(ByVal a as Integer, ByVal b as Integer) as String
Dim target as Integer = CInt(Math.ceiling(a / 10.0)) * 10
IF Math.abs(a-target)>= Math.abs(b-target) Then
return "false"
Else
return "true"
End IF
End Function
This function compares every value with the next based on the row number in order to determine if it must be showed or not. If the value must be showed it returns true, otherwise false string is returned.
You have to pass the value to the function and the next to it, based on the row number. The lookup() function plus row number give us a similar behaviour to a NEXT() function.
=IIf(
Code.Visible(Fields!Value.Value,
Lookup(Fields!RowNumber.Value+1,Fields!RowNumber.Value,Fields!Value.Value,"DataSet11")),
ReportItems!Textbox169.Value,
Nothing
)
For the first row it will pass 3 and 5 to determine if 3 must be showed, then 5 and 6 to determine if the cumulative value must be showed, and so on.
I've created another column as your example, and used the above expression. It says if the function Code.Visible function returns true show the Textbox169 (cumulative value column) otherwise show Nothing.
For Running column I've used the typical running value expression:
=RunningValue(Fields!Value.Value,Sum,"DataSet11")
The result is something like this:
Let me know if this helps.
OK, to set the scene, I have written a function to import multiple tables from MySQL (using RODBC) and run randomForest() on them.
This function is run on multiple databases (as separate instances).
In one particular database, and one particular table, the "error in as.POSIXlt.character(x, tz,.....): character string not in a standard unambiguous format" error is thrown. The function runs on around 150 tables across two databases without any issues except this one table.
Here is a head() print from the table:
MQLTime bar5 bar4 bar3 bar2 bar1 pat1 baXRC
1 2014-11-05 23:35:00 184 24 8 24 67 147 Flat
2 2014-11-05 23:57:00 203 184 204 67 51 147 Flat
3 2014-11-06 00:40:00 179 309 49 189 75 19 Flat
4 2014-11-06 00:46:00 28 192 60 49 152 147 Flat
5 2014-11-06 01:20:00 309 48 9 11 24 19 Flat
6 2014-11-06 01:31:00 24 177 64 152 188 19 Flat
And here is the function:
GenerateRF <- function(db, countstable, RFcutoff) {
'load required libraries'
library(RODBC)
library(randomForest)
library(caret)
library(ff)
library(stringi)
'connection and data preparation'
connection <- odbcConnect ('TTODBC', uid='root', pwd='password', case="nochange")
'import count table and check if RF is allowed to be built'
query.str <- paste0 ('select * from ', db, '.', countstable, ' order by RowCount asc')
row.counts <- sqlQuery (connection, query.str)
'Operate only on tables that have >= RFcutoff'
for (i in 1:nrow (row.counts)) {
table.name <- as.character (row.counts[i,1])
col.count <- as.numeric (row.counts[i,2])
row.count <- as.numeric (row.counts[i,3])
if (row.count >= 20) {
'Delete old RFs and DFs for input pattern'
if (file.exists (paste0 (table.name, '_RF.Rdata'))) {
file.remove (paste0 (table.name, '_RF.Rdata'))
}
if (file.exists (paste0 (table.name, '_DF.Rdata'))) {
file.remove (paste0 (table.name, '_DF.Rdata'))
}
'import and clean data'
query.str2 <- paste0 ('select * from ', db, '.', table.name, ' order by mqltime asc')
raw.data <- sqlQuery(connection, query.str2)
'partition data into training/test sets'
set.seed(489)
index <- createDataPartition(raw.data$baXRC, p=0.66, list=FALSE, times=1)
data.train <- raw.data [index,]
data.test <- raw.data [-index,]
'find optimal trees to grow (without outcome and dates)
data.mtry <- as.data.frame (tuneRF (data.train [, c(-1,-col.count)], data.train$baXRC, ntreetry=100,
stepFactor=.5, improve=0.01, trace=TRUE, plot=TRUE, dobest=FALSE))
best.mtry <- data.mtry [which (data.mtry[,2] == min (data.mtry[,2])), 1]
'compress df'
data.ff <- as.ffdf (data.train)
'run RF. Originally set to 1000 trees but M1 dataset is to large for laptop. Maybe train at the lab?'
data.rf <- randomForest (baXRC~., data=data.ff[,-1], mtry=best.mtry, ntree=500, keep.forest=TRUE,
importance=TRUE, proximity=FALSE)
'generate and print variable importance plot'
varImpPlot (data.rf, main = table.name)
'predict on test data'
data.test.pred <- as.data.frame( predict (data.rf, data.test, type="prob"))
'get dates and name date column'
data.test.dates <- data.frame (data.test[,1])
colnames (data.test.dates) <- 'MQLTime'
'attach dates to prediction df'
data.test.res <- cbind (data.test.dates, data.test.pred)
'force date coercion to attempt negating unambiguous format error '
data.test.res$MQLTime <- format(data.test.res$MQLTime, format = "%Y-%m-%d %H:%M:%S")
'delete row names, coerce to dataframe, generate row table name and export outcomes to MySQL'
rownames (data.test.res)<-NULL
data.test.res <- as.data.frame (data.test.res)
root.table <- stri_sub(table.name, 0, -5)
sqlUpdate (connection, data.test.res, tablename = paste0(db, '.', root.table, '_outcome'), index = "MQLTime")
'save RF and test df/s for future use; save latest version of row_counts to MQL4 folder'
save (data.rf, file = paste0 ("C:/Users/user/Documents/RF_test2/", table.name, '_RF.Rdata'))
save (data.test, file = paste0 ("C:/Users/user/Documents/RF_test2/", table.name, '_DF.Rdata'))
write.table (row.counts, paste0("C:/Users/user/AppData/Roaming/MetaQuotes/Terminal/71FA4710ABEFC21F77A62A104A956F23/MQL4/Files/", db, "_m1_rowcounts.csv"), sep = ",", col.names = F,
row.names = F, quote = F)
'end of conditional block'
}
'end of for loop'
}
'close all connection to MySQL'
odbcCloseAll()
'clear workspace'
rm(list=ls())
'end of function'
}
At this line:
data.test.res$MQLTime <- format(data.test.res$MQLTime, format = "%Y-%m-%d %H:%M:%S")
I have tried coercing MQLTime using various functions including: as.character(), as.POSIXct(), as.POSIXlt(), as.Date(), format(), as.character(as.Date())
and have also tried:
"%y" vs "%Y" and "%OS" vs "%S"
All variants seem to have no effect on the error and the function is still able to run on all other tables. I have checked the table manually (which contains almost 1500 rows) and also in MySQL looking for NULL dates or dates like "0000-00-00 00:00:00".
Also, if I run the function line by line in R terminal, this offending table is processed without any problems which just confuses the hell out me.
I've exhausted all the functions/solutions I can think of (and also all those I could find through Dr. Google) so I am pleading for help here.
I should probably mention that the MQLTime column is stored as varchar() in MySQL. This was done to try and get around issues with type conversions between R and MySQL
SHOW VARIABLES LIKE "%version%";
innodb_version, 5.6.19
protocol_version, 10
slave_type_conversions,
version, 5.6.19
version_comment, MySQL Community Server (GPL)
version_compile_machine, x86
version_compile_os, Win32
> sessionInfo()
R version 3.0.2 (2013-09-25)
Platform: i386-w64-mingw32/i386 (32-bit)
Edit: Str() output on the data as imported from MySQl showing MQLTime is already in POSIXct format:
> str(raw.data)
'data.frame': 1472 obs. of 8 variables:
$ MQLTime: POSIXct, format: "2014-11-05 23:35:00" "2014-11-05 23:57:00" "2014-11-06 00:40:00" "2014-11-06 00:46:00" ...
$ bar5 : int 184 203 179 28 309 24 156 48 309 437 ...
$ bar4 : int 24 184 309 192 48 177 48 68 60 71 ...
$ bar3 : int 8 204 49 60 9 64 68 27 192 147 ...
$ bar2 : int 24 67 189 49 11 152 27 56 437 67 ...
$ bar1 : int 67 51 75 152 24 188 56 147 71 0 ...
$ pat1 : int 147 147 19 147 19 19 147 19 147 19 ...
$ baXRC : Factor w/ 3 levels "Down","Flat",..: 2 2 2 2 2 2 2 2 2 3 ...
So I have tried declaring stringsAsfactors = FALSE in the dataframe operations and this had no effect.
Interestingly, if the offending table is removed from processing through an additional conditional statement in the first 'if' block, the function stops on the table immediately preceeding the blocked table.
If both the original and the new offending tables are removed from processing, then the function stops on the table immediately prior to them. I have never seen this sort of behavior before and it really has me stumped.
I watched system resources during the function and they never seem to max out.
Could this be a problem with the 'for' loop and not necessarily date formats?
There appears to be some egg on my face. The table following the table where the function was stopping had a row with value '0000-00-00 00:00:00'. I added another statement in my MySQL function to remove these rows when pre-processing the tables. Thanks to those that had a look at this.