(BIDS on SQL Server 2008)
I have a flat file (pipe-delimited) which I have successfully parsed to the following format:
AccountID FreeText1 FreeText2 FreeText3 FreeText4
1 Some text More text Other text Different Text
2 Some text More text Other text Different Text
3 Some text More text Other text Different Text
I need the end result to look like this:
AccountID Title TheData
1 FreeText1 Some text
1 FreeText2 More text
1 FreeText3 Other text
1 FreeText4 Different Text
2 FreeText1 Some text
2 FreeText2 More text
2 Freetext3 Other text
2 FreeText4 Different Text
3 FreeText1 Some text
3 FreeText2 More text
3 FreeText3 Other text
3 FreeText4 Different Text
I am still rather new to SSIS so learning as I go. Everything I found on the Unpivot transformation seems to be what I need, but I haven't been able to figure out how to get it to Unpivot based on the NAME of the column ("FreeText1", etc), nor have I been able to fully grasp how to set up the Unpivot transform to even get close to the desired results.
I haven't yet found any SSIS formulas I could use in a Derived Column to get the column name programmatically, thinking maybe I could generate the column names in a Derived Column and then Merge Join the two together... but that doesn't seem like a very efficient method and I couldn't make it work anyway. I have tried setting up a Derived Column to return the column names in hard code (using "FreeText1" as a formula, for example), however I remain unsure as to how to combine this with the Unpivoted results.
Any input would be greatly appreciated!
You could use the UNPIVOT transformation, which should look something like
Or you could load the data to a staging table and use the TSQL UNPIVOT function:
SELECT upvt.AccountID, upvt.Title, upvt.TheData
FROM dbo.StagingTable AS t
UNPIVOT (Title FOR TheData IN (FreeText1, FreeText2, FreeText3, FreeText4)) AS upvt;
Or slightly longer winded, but more flexible is to use CROSS APPLY along with a table value constructor to unpivot data. e.g.
SELECT t.AccountID, upvt.Title, upvt.TheData
FROM dbo.StagingTable AS t
CROSS APPLY
(VALUES
('FreeText1', FreeText1),
('FreeText2', FreeText2),
('FreeText3', FreeText3),
('FreeText4', FreeText4)
) AS upvt (Title, TheData);
I have found that for some people UnPivot is more confusing to learn than it is worth so I have always avoided it when possible. If you want to take a different approach, you can do this:
Load the data into a temp table and run the following query against it:
Select * From
(
Select AccountID, 'FreeText1' Title, FreeText1 TheData From TableA
Union All
Select AccountID, 'FreeText2' Title, FreeText2 TheData From TableA
Union All
Select AccountID, 'FreeText3' Title, FreeText3 TheData From TableA
Union All
Select AccountID, 'FreeText4' Title, FreeText4 TheData From TableA
) A
Order By AccountID, Title
Related
I have table with single row like below which comes from snowflake query like below
show warehouses like 'COMMON_WH';
select "name","state","type","size" FROM TABLE(RESULT_SCAN(LAST_QUERY_ID()))
name|state|type|size
COMMON_WH|SUSPENDED|STANDARD|Small
I Want to rotate it like below.
name| COMMON_WH
state|SUSPENDED
type|STANDARD
size|Small
Thanks,
You can do this with a unpivot. Here is an example of how to use it. Note that it is a bit verbose because you need to cast all of the original table's datatypes to the same datatype before the unpivot (varchar in this case).
show warehouses like 'COMMON_WH';
select *
from (
select
"name"::varchar as name,
"state"::varchar as state,
"type"::varchar as type,
"size"::varchar as size
FROM TABLE (RESULT_SCAN(LAST_QUERY_ID()))
) unpivot (col_val for col_name in (name, state, type, size))
This produces:
COL_NAME
COL_VAL
NAME
COMMON_WH
STATE
STARTED
TYPE
STANDARD
SIZE
X-Small
i am working with json_object where for different product categories we have different color variable for ex- kdt_color,fcy_color, etc. (check below)
How to select appropriate color_variable for each product to extract color value from json_object. There are 100+ verticals so cant use case here.
{"ctg_ideal_for":["Women"],"ctg_fabric":["Chiffon"],"ctg_design_style":["Umbrella Burqa"],"aba_color":["Black"],"aba_sleeve":["Full Sleeves"],"aba_with_hijab":[true]}
{"blz_color":["single"],"blz_size":["34\"36\"38\"40\"42"],"blz_sleeve_type":["Full Sleeves"],"ctg_ideal_for":["Men"],"ctg_fabric":["Imported"],"ctg_design_style":["Plaid Blazer"]}
{"color":["Multicolor"],"material":["PU"],"ideal_for":["Women"],"closure":["Zipper"],"bpk_style_code":["RMMY2418"]}
It is possible to use regexp_extract to extract color":["SomeColor from JSON string like this:
select nbr, regexp_extract(json, 'color":\\["([A-Za-z]*)',1) as color
from
(
select 1 as nbr, '{"ctg_ideal_for":["Women"],"ctg_fabric":["Chiffon"],"ctg_design_style":["Umbrella Burqa"],"aba_color":["Black"],"aba_sleeve":["Full Sleeves"],"aba_with_hijab":[true]}' as json union all
select 2 as nbr, '{"blz_color":["single"],"blz_size":["34\"36\"38\"40\"42"],"blz_sleeve_type":["Full Sleeves"],"ctg_ideal_for":["Men"],"ctg_fabric":["Imported"],"ctg_design_style":["Plaid Blazer"]}' as json union all
select 3 as nbr, '{"color":["Multicolor"],"material":["PU"],"ideal_for":["Women"],"closure":["Zipper"],"bpk_style_code":["RMMY2418"]}' as json
)s;
OK
1 Black
2 single
3 Multicolor
You can also use RegexSerDe to define regex columns in a table DDL.
I have a column in which is stored nothing but text separated by one space. There may be one to maybe 5 words in each field of the column. I need a query to return all the distinct words in that column.
Tried:
SELECT DISTINCT tags FROM documents ORDER BY tags
but does not work.
To Elaborate.
I have a column called tags. In it I may have the following entries:
Row 1 Red Green Blue Yellow
Row 2 Red Blue Orange
Row 3 Green Blue Brown
I want to select all the DISTINCT words in the entire column - all fields. It would return:
Red Green Blue Yellow Orange Brown
If I counted each it would return:
2 Red
2 Green
3 Blue
1 Yellow
1 Brown
1 Orange
To fix this I ended up creating a second table where all keywords where inserted on their own row each along with a record key that tied them back to the original record in the main data table. I then just have to SELECT DISTINCT to get all tags or I can SELECT DISTINCT with a WHERE clause specifying the original record to get the tags associated with a unique record. Much easier.
There is not a good solution for this. You can achieve this with JSON functions as of 5.6, I think, but it's a little tricky until 8.0, when mySQL added the JSON_TABLE function, which can convert json data to a table like object and perform selects on it, but how it will perform is dependent on your actual data. Here's a working example:
CREATE TABLE t(raw varchar(100));
INSERT INTO t (raw) VALUES ('this is a test');
You will need to strip the symbols (commas, periods, maybe others) from your text, then replace any white text with ",", then wrap the whole thing in [" and "] to json format it. I'm not going to give a full featured example, because you know better than I do what your data looks like, but something like this (in its simplest form):
SELECT CONCAT('["', REPLACE(raw, ' ', '","'), '"]') FROM t;
With JSON_TABLE, you can do something like this:
SELECT CONCAT('["', REPLACE(raw, ' ', '","'), '"]') INTO #delimited FROM t;
SELECT *
FROM JSON_TABLE(
#delimited,
"$[*]"
COLUMNS(Value varchar(50) PATH "$")
) d;
See this fiddle: https://dbfiddle.uk/?rdbms=mysql_8.0&fiddle=7a86fcc77408ff5dfec7a805c6e4117a
At this point you have a table of the split words, and you can replace SELECT * with whatever counting query you want, probably SELECT Value, count(*) as vol. You will also need to use group_concat to handle multiple rows. Like this:
insert into t (raw) values ('this is also a test'), ('and you can test it');
select concat(
'["',
replace(group_concat(raw SEPARATOR '","'), ' ', '","'),
'"]'
) into #delimited from t;
SELECT Value, count(*) as vol
FROM JSON_TABLE(
#delimited,
"$[*]"
COLUMNS(Value varchar(50) PATH "$")
) d
GROUP BY Value ORDER BY count(*) DESC;
If you are running <8.0, you can still accomplish this, but it will take some hackiness, like generating an arbitrary list of numbers and constructing the paths dynamically from that.
I have created a table where there are 3 entity id(type varchar), name(type varchar) and menu(type longtext). In the menu I have put some longtxt values, e.g
Now I want to get the id's or the names where the name=any one of the long text vales. I have tried, but simply it didn't succeed.
I think you are looking for FIND_IN_SET:
SELECT * FROM table WHERE FIND_IN_SET(`name`, `menu`);
I have a LONGTEXT column that I use to store paragraphs of text. Let's say that within these paragraphs there is something like:
Text text text text
COUNT: 75 text text
text text text text
text text text text
Would it be possible to do a comparative query the small string "COUNT: 75" out of all that text?
My first thought was something like
SELECT * FROM `books`
WHERE `paragraphs` LIKE '%COUNT: >0%'
Is this even possible?
Your SELECT will only find rows where the text contains exaclty the bit between the wildcard characters: you can't combine a LIKE with comparative logic like that.
What you can do, though, is to strip out the relevant sections of text using a regular expression and then analyse that.
Bear in mind, though, that combining
large amounts of text
textual content logic
regex
all at once will not provide the best performance! I would suggest the following:
use a trigger to strip out a subsection of text so that you have
something manageable (i.e. 50 characters or so) to work with,
inserting this subtext into a separate table
use MySql regex or fulltext functions to analyse your COUNTs
So your trigger would have something like:
select
ltrim(rtrim(substring(paragraphs, instr(paragraphs, 'count:') + 6, 10)))
from books
where instr(paragraphs, 'count:') > 0
which would get you the next 10 characters after 'count:', trimmed of whitespace. you could then further refine that by e.g.
select substring(text_snippet, 1, instr(text_snippet, ' ')) as count_value
from
(
select
ltrim(rtrim(substring(paragraphs, instr(paragraphs, 'count:') + 6, 10)))
as text_snippet
from books
where instr(paragraphs, 'count:') > 0
) x
where isnumeric(substring(text_snippet, 1, instr(text_snippet, ' '))) = 1
to get rows where a numerical value follows the COUNT bit.
You can then extract numerical values next to COUNT, saving them as numbers in a separate table, and then use a JOIN like this:
select b.*
from books b inner join books_count_values v
on b.id = v.books_id
where v.count_value > 0
See here
http://dev.mysql.com/doc/refman/5.1/en/string-functions.html
for the functions at your disposal.