I have CSV file with comma separator in the below format,
Sensor_1
Val1
sensor_2
Val2
22-02-22
2344
22-02-22
4322
23-02-22
6738
23-02-22
3773
I want to convert it to the below format
Sensor_V
Timestam
Valu
Sensor_1
22-02-22
2344
Sensor_1
23-02-22
6738
Sensor_2
22-02-22
4322
Sensor_2
23-02-22
3773
I am trying it in SSIS, I tried unpivot operation and transpose nothing is working, Can anyone please help me to find a way to convert the file and import to sql table
Thanks in advance!
Related
I have a hive table with a date field in it.
+----------+------+-----+
|data_field| col1| col2|
+----------+------+-----+
|10/01/2018| 125| abc|
|10/02/2018| 124| def|
|10/03/2018| 127| ghi|
|10/04/2018| 127| klm|
|10/05/2018| 129| nop|
+----------+------+-----+
I am reading the table as below.
hive_context = HiveContext(sc)
df = hive_context.sql("select data_field, col1 , col2 from table")
I would like to split the input data into several files based on the date_field column and drop it in the date_field folder. The output should look something like below.
/data/2018-10-01/datafile.csv
/data/2018-10-02/datafile.csv
/data/2018-10-03/datafile.csv
/data/2018-10-04/datafile.csv
/data/2018-10-05/datafile.csv
for example: The file (/data/2018-10-01/datafile.csv ) should have the below data.
data_field,col1,col2
10/01/2018,125,abc
What approach should I take to achieve this?
Look at partitionBy() in DataFrameWriter class. Example usage would be df.write.partitionBy(date)...
I need to store data in a database. I have installed and configured a MySQL database (and an SQLite database) in Matlab. However I cannot store and retrieve anything other than scalar numeric values.
% create an empty database called test_data base with MySQL workbench.
% connect to it in Matlab
conn=database('test_database','root','XXXXXX','Vendor','MySQL');
% create a table to store values
create_test_table=['CREATE TABLE test_table (testID NUMERIC PRIMARY KEY, test_string VARCHAR(255), test_vector BLOB, test_scalar NUMERIC)'];
curs=exec(conn,create_test_table)
Result is good so far (curs.Message is an empty string)
% create a new record
datainsert(conn,'test_table',{'testID','test_string','test_vector','test_scalar'},{1,'string1',[1,2],1})
% try to read out the new record
sqlquery='SELECT * FROM test_table8';
data_to_view=fetch(conn,sqlquery)
Result is bad:
data_to_view =
1 NaN NaN 1
From the documentation for "fetch" I would expect:
data_to_view =
1×4 table
testID test_string test_vector test_scalar
_____________ ___________ ______________ ________
1 'string1' 1x2 double 1
Until I learn how to read blobs I'd even be willing to accept:
data_to_view =
1×4 table
testID test_string test_vector test_scalar
_____________ ___________ ______________ ________
1 'string1' NaN 1
I get the same thing with an sqlite database. How can I store and then read out strings and blobs and why isn't the data returned in table format?
Matlab does not document that the default options for SQLite and MySQL database retrieval are to attempt to return everything as a numeric array. One only needs this line:
setdbprefs('DataReturnFormat','cellarray')
or
setdbprefs('DataReturnFormat','table')
in order to get results with differing datatypes. However! now my result is:
data_to_view =
1×4 cell array
{[2]} {'string1'} {11×1 int8} {[1]}
If instead I input:
datainsert(conn,'test_table',{'testID','test_string','test_vector','test_scalar'},{1,'string1',typecast([1,2],'int8'),1})
Then I get:
data_to_view =
1×4 cell array
{[2]} {'string1'} {16×1 int8} {[1]}
which I can convert like so:
typecast(data_to_view{3},'double')
ans =
1 2
Unfortunately this does not work for SQLite. I get:
data_to_view =
1×4 cell array
{[2]} {'string1'} {' �? #'} {[1]}
and I can't convert the third part correctly:
typecast(unicode2native(data_to_view{1,3}),'double')
ans =
0.0001 2.0000
So I still need to learn how to read an SQLite blob in Matlab but that is a different question.
I have a text file S_data.txt with four columns and a typical row is like:
13 Kate 2.138 8/13/2001 13:24:33 (columns are separated by tab)
I want to load the data into a table s_table with four fields: S_id MEDIUMINT, S_name VARCHA(20), S_value DOUBLE, S_dt DATETIME.
mysql>LOAD DATA LOCAL INFILE 'C:\\temp\\S_data.txt' INTO TABLE s_table
LINES TERMINATED BY '\r\n' SET S_dt = STR_TO_DATE(#S_dt,'%m/%d/%y %H:%i:%s');
The values of S_dt all become 0000-00-00 00:00:00.
Can someone help? Thanks.
I found the problem. For my datetime string (e.g. 8/13/2001 13:23:56), I have to use format '%c/%d/%Y %H:%i%S'. Thanks everyone.
SQL Server 2008 and Crystal Reports XI
I'm not exactly sure why I can't convert the values. I have tried cast and convert which both result in the same error. This column did initially contain alphanumeric values but after my primary query inserts values into a temp table there are only numbers in there.
The conversion
CASE WHEN ISNUMERIC(myfield) = 1 THEN CAST(myfield AS INT) ELSE 0 END
returns zeros. An example of the data in the column is:
65
89
151
175
210
I'm sure I'm missing something obvious but I can't figure it out. In crystal reports I was able to successfully use:
IF isnumeric({myfield}) then tonumber({myfield}) else 999
That converts successfully without any issues. Why would it work in crystal and not in SQL?
The answer to this was there was a hidden carriage return in the varchar column so I used:
REPLACE(myfield, CHAR(13), ''),
and was able to convert it without any trouble.
I am trying to write a bash script that will take a text file and use its content to populate a MySQL table.
The text file contains numerous entries that I will need to add to the table.
Example of text file contents:
Firstname=bob
Surname=ross
Age=9
Firstname=gary
Surname=graeme
Age=19
Firstname=henry
Surname=harry
Age=23
The text file is structured like the example above, the same 3 variables are constantly assigned new values for each unique entry that would go into the mysql table.
I have used sed to remove all empty lines and lines beginning with "#" in the text file that I would not want to process.
My question is how do I use bash to make an sql insert every three lines using the contents in textfile to generate the mysql query. I know I need to loop through the file 3 lines at a time until the end of the text file is reached and somehow parse each value after "="
and assign it to a variable in the loop iteration to insert the mysql data but I am not sure how to do this.
I know that I will need to have a new table entry every 3 lines.
echo "INSERT INTO $table (Firstname, Surname, Age) VALUES ('$Firstname', '$Surname', '$age');" | mysql $db
Thanks
I split each line in the input on = and use the second part.
Then I pipe this to the while-loop where I use read fname sname age to populate the variables inside the loop. I use NR % 3==0 {print("")} to introduce a linebreak in the output every three lines.
awk -F"=" '{printf("%s ", $2)}; NR % 3==0 {print("")}' input_file | \
while read fname sname age; do
echo "INSERT INTO $table (Firstname, Surname, Age)
VALUES ('$fname', '$sname', '$age');" | mysql $db
done
UPDATED:
input:
Firstname=bob
Surname=ross
Age=9
Activity=1 2 3 4 5
Firstname=gary
Surname=graeme
Age=19
Activity=1 2 3 4 5
Firstname=henry
Surname=harry
Age=23
Activity=1 2 3 4 5
with this code:
awk -F"=" '{printf("%s ", $2)}; NR % 4==0 {print("")}' input | \
while read fname sname age act; do
echo "'$fname', '$sname', '$age', '$act');"
done
gives:
'bob', 'ross', '9', '1 2 3 4 5');
'gary', 'graeme', '19', '1 2 3 4 5');
'henry', 'harry', '23', '1 2 3 4 5');
If it's possible to change the structure of file, just try to use LOAD DATA INFILE ... instruction. Detailed specification is here. It's a lot faster than you would write it by yourself :)
If not, try this.