I am trying to pull a product code from a long set of string formatted like a URL address. The pattern is always 3 letters followed by 3 or 4 numbers (ex. ???### or ???####). I have tried using REGEXP and LIKE syntax, but my results are off for both/I am not sure which operators to use.
The first select statement is close to trimming the URL to show just the code, but oftentimes will show a random string of numbers it may find in the URL string.
The second select statement is more rudimentary, but I am unsure which operators to use.
Which would be the quickest solution?
SELECT columnName, SUBSTR(columnName, LOCATE(columnName REGEXP "[^=\-][a-zA-Z]{3}[\d]{3,4}", columnName), LENGTH(columnName) - LOCATE(columnName REGEXP "[^=\-][a-zA-Z]{3}[\d]{3,4}", REVERSE(columnName))) AS extractedData FROM tableName
SELECT columnName FROM tableName WHERE columnName LIKE '%___###%' OR columnName LIKE '%___####%'
-- Will take a substring of this result as well
Example Data:
randomwebsite.com/3982356923abcd1ab?random_code=12480712_ABC_DEF_ANOTHER_CODE-xyz123&hello_world=us&etc_etc
In this case, the desired string is "xyz123" and the location of said pattern is variable based on each entry.
EDIT
SELECT column, LOCATE(column REGEXP "([a-zA-Z]{3}[0-9]{3,4}$)", column), SUBSTR(column, LOCATE(column REGEXP "([a-zA-Z]{3}[0-9]{3,4}$)", column), LENGTH(column) - LOCATE(column REGEXP "^.*[a-zA-Z]{3}[0-9]{3,4}", REVERSE(column))) AS extractData From mainTable
This expression is still not grabbing the right data, but I feel like it may get me closer.
I suggest using
REGEXP_SUBSTR(column, '(?<=[&?]random_code=[^&#]{0,256}-)[a-zA-Z]{3}[0-9]{3,4}(?![^&#])')
Details:
(?<=[&?]random_code=[^&#]{0,256}-) - immediately on the left, there must be & or &, random_code=, and then zero to 256 chars other than & and # followed with a - char
[a-zA-Z]{3} - three ASCII letters
[0-9]{3,4} - three to four ASCII digits
(?![^&#]) - that are followed either with &, # or end of string.
See the online demo:
WITH cte AS ( SELECT 'randomwebsite.com/3982356923abcd1ab?random_code=12480712_ABC_DEF_ANOTHER_CODE-xyz123&hello_world=us&etc_etc' val
UNION ALL
SELECT 'randomwebsite.com/3982356923abcd1ab?random_code=12480712_ABC_DEF_ANOTHER_CODE-xyz4567&hello_world=us&etc_etc'
UNION ALL
SELECT 'randomwebsite.com/3982356923abcd1ab?random_code=12480712_ABC_DEF_ANOTHER_CODE-xyz89&hello_world=us&etc_etc'
UNION ALL
SELECT 'randomwebsite.com/3982356923abcd1ab?random_code=12480712_ABC_DEF_ANOTHER_CODE-xyz00000&hello_world=us&etc_etc'
UNION ALL
SELECT 'randomwebsite.com/3982356923abcd1ab?random_code=12480712_ABC_DEF_ANOTHER_CODE-aaaaa11111&hello_world=us&etc_etc')
SELECT REGEXP_SUBSTR(val,'(?<=[&?]random_code=[^&#]{0,256}-)[a-zA-Z]{3}[0-9]{3,4}(?![^&#])') output
FROM cte
Output:
I'd make use of capture groups:
(?<=[=\-\\])([a-zA-Z]{3}[\d]{3,4})(?=[&])
I assume with [^=\-] you wanted to capture string with "-","\" or "=" in front but not include those chars in the result. To do that use "positive lookbehind" (?<=.
I also added a lookahead (?= for "&".
If you'd like to fidget more with regex I recommend RegExr
I have a varchar(255) field within a source table and the following contents:
50339 My great example
2020002 Next ID but different title
202020 Here we go
Now I am processing the data and do an insert select query on it. From this field I would need the INT number at the beginning of the field. IT IS followed by 2 spaces and a text with var length, this text is what I need as well but for another field. In General I want to to put text and ID in two fields which are now in one.
I tried to grab it like this:
SELECT STATUS REGEXP '^(/d{6,8}) ' FROM products_test WHERE STATUS is not null
But then I learned that in MySQL 5.x there are no regexp within the SELECT statement.
How could I seperate those values within a single select statment, so I can use it in my INSERT SELECT?
From the correct solution of user slaakso, resulted another related problem since somtimes the STATUS field is empty which then results in only one insert, but in case there is a value I split it into two fields. So the count does not match.
My case statement with his solution somehow contains a syntax problem:
CASE STATUS WHEN ''
THEN(
NULL,
NULL
)
ELSE(
cast(STATUS as unsigned),
substring(STATUS, locate(' ', STATUS)+3)
)
END
You can do following. Note that you need to treat the columns separately:
select
if(ifnull(status, '')!='', cast(status as unsigned), null),
if(ifnull(status, '')!='', substring(status, locate(' ', status)+2), null)
from products_test;
See db-fiddle
(This is an extension to this question, but my reputation is too low to comment or ask more questions on that topic...)
We work on bigquery, hence limited in importing packages or using other languages. And, as per the link above, js is a solution, but not what I'm looking for here. I implemented it in js, and it was too slow for our needs.
Suppose one of our columns is a string that look like this (array of json):
[{"location":[22.99902,66.000],"t":1},{"location":[55.32168,140.556],"t":2},{"location":[85.0002,20.0055],"t":3}]
I want to extract from the column the json for which "t":2
Where:
some columns don't have elements "t":2
Some columns have several elements "t":2
The number of json elements in each string can change
element "t":2 is not always in second position.
I don't know regexp well enough for this. We tried regexp_extract with this pattern: r'(\{.*?\"t\":2.*?\})')), but that doesn't work. It extracts everything that precedes "t":2, including the json for "t":2. We only want the json of element "t":2.
Could you advise a regexp pattern that would work?
EDIT:
I have a preference for a solution that gives me 1 match. Suppose I have this string:
[{"location":[22.99902,66.000],"t":1},{"location":[55.32168,140.556],"t":2},{"location":[55.33,141.785],"t":2}],
I would prefer receiving only 1 answer, the first one.
In that case perhaps regexp is less appropriate, but I'm really not sure?
How about this:
(?<=\{)(?=.*?\"t\"\s*:\s*2).*?(?=\})
As seen here
There is another solution but it is not regexp based (as I had originally asked). So this should not count as the final answer to my own question, nonetheless could be useful.
It is based on a split of the string in array and then chosing the element in the array that satisfies my needs.
Steps:
transform the string into something better for splits (using '|' as seperator):
replace(replace(replace(my_field,'},{','}|{'),'[{','{'),'}]','}')
split it using split(), which yields an array of strings (each one a json element)
find the relevant element ("t":2) - in my case, the first one is good enough, so I limit the query to 1: array( select data from unnest(split(replace(replace(replace(my_field,'},{','}|{'),'[{','{'),'}]','}'),'|')) as data where data like '%"t":2%' limit 1)
Convert that into a useable string with array_to_string() and use json_extract on that string to extract the relevant info from the element that I need (say for example, location coordinate x).
So putting it all together:
round(safe_cast(json_extract(array_to_string(array( select data from unnest(split(replace(replace(replace(my_field,'},{','}|{'),'[{','{'),'}]','}'),'|')) as data where data like '%"t":2%' limit 1),''),'$.location[0]') as float64),3) loc_x
May 1st, 2020 Update
A new function, JSON_EXTRACT_ARRAY, has been just added to the list of JSON
functions. This function allows you to extract the contents of a JSON document as
a string array.
so in below you can replace use of json2array UDF with just in-built function JSON_EXTRACT_ARRAY as in below example
#standardSQL
SELECT id,
(
SELECT x
FROM UNNEST(JSON_EXTRACT_ARRAY(json, '$')) x
WHERE JSON_EXTRACT_SCALAR(x, '$.t') = '2'
) extracted
FROM `project.dataset.table`
==============
Below is for BigQuery Standard SQL
#standardSQL
CREATE TEMP FUNCTION json2array(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
return JSON.parse(json).map(x=>JSON.stringify(x));
""";
SELECT id,
(
SELECT x
FROM UNNEST(json2array(JSON_EXTRACT(json, '$'))) x
WHERE JSON_EXTRACT_SCALAR(x, '$.t') = '2'
) extracted
FROM `project.dataset.table`
You can test, play with above using dummy data as in below example
#standardSQL
CREATE TEMP FUNCTION json2array(json STRING)
RETURNS ARRAY<STRING>
LANGUAGE js AS """
return JSON.parse(json).map(x=>JSON.stringify(x));
""";
WITH `project.dataset.table` AS (
SELECT 1 id, '[{"location":[22.99902,66.000],"t":1},{"location":[55.32168,140.556],"t":2},{"location":[85.0002,20.0055],"t":3}]' json UNION ALL
SELECT 2, '[{"location":[22.99902,66.000],"t":11},{"location":[85.0002,20.0055],"t":13}]'
)
SELECT id,
(
SELECT x
FROM UNNEST(json2array(JSON_EXTRACT(json, '$'))) x
WHERE JSON_EXTRACT_SCALAR(x, '$.t') = '2'
) extracted
FROM `project.dataset.table`
with output
Row id extracted
1 1 {"location":[55.32168,140.556],"t":2}
2 2 null
Above assumes that there is no more than one element with "t":2 in json column. In case if there can be more than one - you should add ARRAY as below
SELECT id,
ARRAY(
SELECT x
FROM UNNEST(json2array(JSON_EXTRACT(json, '$'))) x
WHERE JSON_EXTRACT_SCALAR(x, '$.t') = '2'
) extracted
FROM `project.dataset.table`
Even though, you have posted a work around your issue. I believe this answer will be informative. You mentioned that one of the answer selected more than what you needed, I wrote the query below to reproduce your case and achieve aimed output.
WITH
data AS (
SELECT
" [{ \"location\":[22.99902,66.000]\"t\":1},{\"location\":[55.32168,140.556],\"t\":2},{\"location\":[85.0002,20.0055],\"t\":3}] " AS string_j
UNION ALL
SELECT
" [{ \"location\":[22.99902,66.000]\"t\":1},{\"location\":[55.32168,140.556],\"t\":3},{\"location\":[85.0002,20.0055],\"t\":3}] " AS string_j
UNION ALL
SELECT
" [{ \"location\":[22.99902,66.000]\"t\":1},{\"location\":[55.32168,140.556],\"t\":3},{\"location\":[85.0002,20.0055],\"t\":3}] " AS string_j
UNION ALL
SELECT
" [{ \"location\":[22.99902,66.000]\"t\":1},{\"location\":[55.32168,140.556],\"t\":3},{\"location\":[85.0002,20.0055],\"t\":3}] " AS string_j ),
refined_data AS (
SELECT
REGEXP_EXTRACT(string_j, r"\{\"\w*\"\:\[\d*\.\d*\,\d*\.\d*\]\,\"t\"\:2\}") AS desired_field
FROM
data )
SELECT
*
FROM
refined_data
WHERE
desired_field IS NOT NULL
Notice that I have used the dummy described in the temp table, populated inside the WITH method. As below:
Afterwords, in the table refined_data, I used the REGEXP_EXTRACT to extract the desired string from the column. Observe that for the rows which there is not a match expression, the output is null. Thus, the table refined_data is as follows :
As you can see, now it is just needed a simple WHERE filter to obtain the desired output, which was done in the last select.
In addition you can see the information about the regex expression I provided here.
I tried something out. Here is a simple example in SQL Fiddle: Example
There is a column someNumbers (comma-seperated numbers) and I tried to get all the rows where this column contains a specific number. Problem is, the result only contains rows where someNumbers starts with the specific number.
The query SELECT * FROM myTable where 2 in ( someNumbers ) only returns the row with id 2 and not the row with id 1.
Any suggestions? Thank you all.
You are storing data in the wrong format! You should not be storing multiple values in a single string column. You should not be storing numbers as strings. Instead, you should have a junction table with one row per id and per number.
Sometimes, you just have no choice, because someone else created a really poorly designed database. For these situations, MySQL has the function find_in_set():
SELECT *
FROM myTable
WHERE find_in_set(2, someNumbers ) > 0;
The right solution, however, is to fix the data model.
While Gordon's answer is a good one, here is a way to do this with like
SELECT * FROM myTable where someNumbers like '2,%' or someNumbers like '%,2,%' or someNumbers like '%,2'
The first like checks if your array starts with the number you are looking for (2). The second one checks if 2 is within the array and the last like tests for appearance at the end.
Note that the commas are essential here, because something like '%2%' would also match ...,123,...
EDIT: As suggested by the OP it may happen that only a single value is present in the row. Consequently, the query must check this case by doing ... someNumbers = '2'
I would suggest this query :
SELECT * FROM myTable where someNumbers like '%2%'
It will select every entry where someNumbers contains '2'
Select * from table_name where coloumn_name IN(value,value,value)
you can use it
I have two select statements which are as mentioned below
SELECT 'MY OUTPUT'
SELECT * FROM MY TABLE
On execution MY OUTPUT is printed first and then there is a gap for next select.
I want to use something like UNION to combine two statements.
I'm using:
SELECT 'MY OUTPUT' UNION
SELECT * FROM MY TABLE
But, I am getting error:
All queries combined using a UNION, INTERSECT or EXCEPT operator must have an equal number of expressions in their target lists.
SELECT 'MY OUTPUT' # this query return one column
UNION
SELECT * FROM MY TABLE # this query return more than one column
number of columns must be the same
SELECT 'MY OUTPUT' UNION # return one column
SELECT column1 FROM MY TABLE # return one column now it will work
but i think you want to do this no?
SELECT 'MY OUTPUT',column1,Column2,column3 FROM MY TABLE
So here's what's happening:
The error you're getting (All queries combined using a UNION, INTERSECT or EXCEPT operator must have an equal number of expressions in their target lists.) is because there are more columns in one select than the other.
You could use the same blank columns in your first query as there are in [My Table] which would make your query look like:
SELECT 'MY OUTPUT' , '', '' ,'', '' --(no. of columns should match those in MY TABLE)
UNION
SELECT * FROM [MY TABLE]
I'm guessing you want an excel style cell merge which is not possible as the output of a select query unfortunately.