I'm trying to export a MySQL table into csv file using SequelPro, however the result file is not properly formatted. For instance, on the first column should the id, in some of the records there is a number, but in others there is text instead. This happens in pretty much every column.
One explanation might be that the field are being exported vertically rather than horizontally (you have the id 1 on the first row and the id 10 on the 10th row.
Maybe I'm getting the the options (fields) wrong, in which case I'd appreciate help.
What I need is for the file to have the titles as the first row and the actual data in the subsequent rows with each data property as a column.
Appreciate the help
PS: I've tried this but I'm getting the data vertically as well.
For convenience in executing and sharing across the team, consider adding commas as the part of SELECT statement itself like this
SELECT
col1 as 'Column 1',',',
col2 as 'Column 2',',',
col3 as 'Column 3',',',
col4 as 'Column 4',',',
col5 as 'Column 5',','
FROM
table1, table2
WHERE
clause1 and clause2
The data and headers do automatically get their necessary commas.
Related
I have created an external Hive table from a tweets json file which is exported from Mongo DB. Whenever I select more than one column from the hive table, the retrieved results are not well formatted. some columns are empty or NULL (even if i conditioned on specific values)and some data appear in the wrong columns.
I think this is happening because the text has commas in it. when i try to query the hive table without selecting the text of the tweets, the results make sense then. But i don't know how to fix that.
Anyone Has any idea how to fix that??
Best,
Why dont you try formatting the output? Something like this -
SELECT
CONCAT(COALESCE(COL1,''),
'|', COALESCE(COL2,''),
'|', COALESCE(COL3,''),
'|', COALESCE(COL4,''),
'|', COALESCE(COL5,''),
'|', COALESCE(COL6,''),
'|', COALESCE(COL7,'')) as tweetsout
FROM (
SELECT COL1, COL2, COL3, COL4, COL5, COL6, COL7
FROM TWEETS
) TOUT
This would give you the output delimied by a pipe instead of the standard tab delimited output.
It is difficult to tell without knowing the exact create table command you used...
Usually the table is parsed incorrectly if the input data contains table delimiters. For example, some tweets in your input database may contain \n which might be a row separator in the hive table you created.
I have 2 identical tables (both column count, name and settings).
I have 1 identical row on each table, identical information.
One of the columns is a BLOB type column and contains an Image (bmp).
Both rows / tables have an id column, auto increment, so id's are identical for every row in both tables.
The column set to blob type CAN be NULL (it's set).
I'm using an action that has the following query's:
dbmodule.arhivaQuery.SQL.Clear;
dbmodule.arhivaQuery.SQL.Add('UPDATE `database_name`.`2nd_table_name` SET `column_name`=deleted WHERE `id`=''' + inttostr(dbmodule.comenziDataSetid.Value) + ''';');
dbmodule.arhivaQuery.ExecSql(true);
This should theoretically update the row in the 2nd table by removing the bmp from the blob column, or rather, replacing the bmp with the word "deleted". It doesn't, the image is still in the column / row.
A few things to clarify:
dbmodule is the name of a data module that contains the dataset, data source, sql connection and query. (TSimpleDataSet, TDataSource, TSQLConnection, TSQLQuery).
arhivaQuery is the name of the query I'm using (a TSQLQuery)
column_name = name of the column, I've edited in this paste so you can get a clearer picture.
You notice at the end I use the id of the row from the 1st table to change the data in the 2nd, so that's why that is there (ids are identical for the row in both tables).
When I execute this it should keep both rows in both tables but update just the row in the 2nd table by removing its image from the blob column.
So after this I should have row in 1st table with the image in the blob column and same row in the 2nd table with no image in the blob column.
I'm guessing my sql syntax is wrong (got some warnings saying so too), can anyone correct it for me please?
If you want to clear the content of a blob field, assign it the value NULL.
dbmodule.arhivaQuery.SQL.Add('UPDATE `database_name`.`2nd_table_name` SET `column_name` = NULL WHERE `id`=''' + inttostr(dbmodule.comenziDataSetid.Value) + ''';');
If you want to display deleted for those columns that have no value, do that in your SELECT statement when retrieving the content using IFNULL() or COALESCE(), whichever your DBMS supports.
An additional improvement you could make (both for coding ease and prevention of SQL injection) is to stop concatenating your SQL and switch to using parameters. It also means you can stop with all of the ''' double/triple/quadruple quoting nonsense and data type conversions, because the DB driver will take care of all of that for you.
dbmodule.arhivaQuery.SQL.Add('UPDATE `database_name`.`2nd_table_name` SET `column_name` = NULL WHERE `id`= :ID;');
// Use AsInteger, AsString, AsFloat, or AsBoolean, whichever fits your
// column data type. Notice no quotes, no call to IntToStr or FloatToStr.
dbmodule.arhivaQuery.ParamByName('ID').AsString := dbmodule.comenziDataSetid.Value;
NOTE: Some DB drivers will need Params.ParamByName instead. If one doesn't work, the other will.
Finally, break your long lines of SQL into manageable pieces so you can stop all of the scrolling around to read it.
dbmodule.arhivaQuery.SQL.Add('UPDATE `database_name`.`2nd_table_name`');
dbmodule.arhivaQuery.SQL.Add('SET `column_name` = NULL WHERE `id`= :ID;');
dbmodule.arhivaQuery.ParamByName('ID').AsString := dbmodule.comenziDataSetid.Value;
I have downloaded a MySQL table as CSV, which has over thousand entries of the following type:
id,gender,garment-color
1,male,white
2,"male,female",black
3,female,"red,pink"
Now, when I am trying to create a chart out of this data, it is taking "male" as one value, and "male,female" as a separate value.
So, for the above example, rather than counting 2 "male", and 3 "female", the chart is showing 3 separate categories ("male", "female", "male,female"), with one count each.
I want the output as follows, for chart to have the correct count:
id,gender,garment-color
1,male,white
2,male,black
2,female,black
3,female,red
3,female,pink
The only way I know is to copy the row in MS Excel and adjust the values manually, which is too tedious for 1000+ entries. Is there a better way?
From MySQL command line or whatever tool you are using to send queries to MySQL:
select * from the_table
into outfile '/tmp/out.txt' fields terminated by ',' enclosed by '"'
Then download /tmp/out.txt' from the server and it should be good to go assuming your data is good. If it is not, you might need to massage it with some SQL function use in theselect`.
The csv likely came from a poorly designed/normalized database that had both those values in the same row. You could try using selects and updates, along some built in string functions, on such rows to spawn additional rows containing the additional values and update their original rows to remove those values; but you will have to repeat until all commas are removed (if there is more than one in some field), and will have to determine if a row containing multiple fields with such comma-separated lists need multiplied out (i.e. should 2 gender and 4 color mean 8 rows total).
More likely, you'll probably want to create additional tables for X_garmentcolors, and X_genders; where X is whatever the original table is supposed to be describing. These tables would have an X_id field referencing the original row and a [garmentcolor|gender] value field holding one of the values in the original rows lists. Ideally, they should actually reference [gender|garmentcolor] lookup tables instead of holding actual values; but you'd have to do the grunt work of picking out all the unique colors and genders from your data first. Once that is done, you can do something like:
INSERT INTO X_[garmentcolor|gender] (X_id, Y_id)
SELECT X.X_id, Y.Y_id
FROM originalTable AS X
INNER JOIN valueTable AS Y
ON X.Y_valuelist LIKE CONCAT('%,' Y.value) -- Value at end of list
OR X.Y_valuelist LIKE CONCAT('%,' Y.value, ',%') -- Value in middle of list
OR X.Y_valuelist LIKE CONCAT(Y.value, ',%') -- Value at start of list
OR X.Y_valuelist = Y.value -- Value is entire list
;
I'm new to Kettle, but getting on well with it so far. However I can't figure out how to do this.
I have a csv which looks something like this
a, col1, col2, col3
a, col1, col2, col3
a, col1, col2, col3
b, col1, col2, col3
b, col1, col2, col3
c, col1, col2, col3
c, col1, col2, col3
The first column starts with a key (a,b,c), and then the rest of the columns follow. What I want to do is read in the csv (got that covered) and then split the csv based on key, so I have 3 chunks/ groups of data and then convert each of those chunks of data into a separate json file, which I think I can get.
What I can't get my head around is the grouping the data and then performing a separate action (convert to json) on each of those separate groups. Its not the creating json I have an issue with.
The data is from a sensor network of many environmental sensors so there are many keys, hundreds, and new ones get added. I've used map reduce to process this data before as the concept of partitioning is what I'm trying to replicate here, without using the hadoop elements of kettle as the deployment is different. Once I've partitioned the data it needs to be loaded into different places as seperate records. The key is a unique ID (serial number) of a sensor.
Any ideas please?
Thanks
I guess create a javascript to output the fields of a row in a JSON like string added to the row:
{"id":"a","col1":"1","col2":"2","col3":"3"}
Next you could use the group step and set the base field to the 'id' field and have as aggregate the javascript value in type 'Concatenate strings separated by ,'
{"id":"a","col1":"1","col2":"2","col3":"3"},{"id":"a","col1":"4","col2":"5","col3":"6"}, {"id":"a","col1":"7","col2":"8","col3":"9"}
Add some tags around it and you have valid json. Next you could assemble a file name using javascript step:
var file_name="C:\\dir\\"+ id + ".txt";
Use the text file output and set the file name field to 'file_name'. Remove separator / enclosure options to have none extra formatting and you are done.
If i have understood your question correctly, you can use "GROUP BY" step to group the columns (i.e. the first header in your data set) and then store these into memory.
Once this is done.. use parameter looping to "get the variables" and dynamically generate multiple JSON output. Check the image below:
In the JSON output step, use variables like header1 to generate multiple files. Highlighted below the changes i made in the JSON Output.
In case you find in confusing, i have uploaded a sample code in here.
Hope it helps :)
I have a table with a field where words are written separated with new lines. So a select on this single field from to rows will output 3 lines for first row and 2 lines for second row:
Row1 designationer
nye kolonier
mindre byer
Row2 udsteder
bopladser
I would like to do a select that select all these lines as if they had been rows in the table like:
SELECT do_the_split(field) FROM table
so the result would be more like:
Row1 designationer
Row2 nye kolonier
Row3 mindre byer
Row4 udsteder
Row5 bopladser
is there any way to do this in MySQL?
BR. Anders
UPDATE: There are correct answers below but ended up doing it in a semi-manual way. I did this:
exporting the the column to a csv
Open file in notepad and remove the pings around each line
now each word is on a new line
save as csv
import into database where each line will be a new row
Br. Anders
You can use a stored procedure - similar to what this person did - to accomplish this, essentially utilizing a temp table.
Certainly you could accomplish this locally in your app, as MasterPeter has suggested.
I've faced the same problem and the only two ways I know of getting the kind of collection of words you want are stored procedures (which is what I did, although with the Derby DB) or a script/program.
I don't know the in-and-outs of mySQL to be of any help in that but what's stopping you from doing the splitting in the application layer? Say, you load your rows, as they are, in Java (or PHP or whatever) and then
String row = <fetch row from resultset>;
String[] individualItems = row.split("\n");
If you could store the values like that in your DB, you could just as well retrieve them like that.