PostgreSQL function with a loop - function

I'm not good at postgres functions. Could you help me out?
Say, I have this db:
name | round |position | val
-----------------------------------
A | 1 | 1 | 0.5
A | 1 | 2 | 3.4
A | 1 | 3 | 2.2
A | 1 | 4 | 3.8
A | 2 | 1 | 0.5
A | 2 | 2 | 32.3
A | 2 | 3 | 2.21
A | 2 | 4 | 0.8
I want to write a Postgres function that can loop from position=1 to position=4 and calculate the corresponding value. I could do this in python with psycopg2:
import psycopg2
import psycopg2.extras
conn = psycopg2.connect("host='localhost' dbname='mydb' user='user' password='pass'")
CURSOR = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
cmd = """SELECT name, round, position, val from mytable"""
CURSOR.execute(cmd)
rows = CURSOR.fetchall()
dict = {}
for row in rows:
indx = row['round']
try:
dict[indx] *= (1-row['val']/100)
except:
dict[indx] = (1-row['val']/100)
if row['position'] == 4:
if indx == 1:
result1 = dict[indx]
elif indx == 2:
result2 = dict[indx]
print result1, result2
How can I do the same thing directly in Postgres so that it returns a table of (name, result1, result2)
UPDATE:
#a_horse_with_no_name, the expected value would be:
result1 = (1 - 0.5/100) * (1 - 3.4/100) * (1 - 2.2/100) * (1 - 3.8/100) = 0.9043
result2 = (1 - 0.5/100) * (1 - 32.3/100) * (1 - 2.21/100) * (1 - 0.8/100) = 0.6535

#Glenn gave you a very elegant solution with an aggregate function. But to answer your question, a plpgsql function could look like this:
Test setup:
CREATE TEMP TABLE mytable (
name text
, round int
, position int
, val double precision
);
INSERT INTO mytable VALUES
('A', 1, 1, 0.5)
, ('A', 1, 2, 3.4)
, ('A', 1, 3, 2.2)
, ('A', 1, 4, 3.8)
, ('A', 2, 1, 0.5)
, ('A', 2, 2, 32.3)
, ('A', 2, 3, 2.21)
, ('A', 2, 4, 0.8)
;
Generic function
CREATE OR REPLACE FUNCTION f_grp_prod()
RETURNS TABLE (name text
, round int
, result double precision)
LANGUAGE plpgsql STABLE AS
$func$
DECLARE
r mytable%ROWTYPE;
BEGIN
-- init vars
name := 'A'; -- we happen to know initial value
round := 1; -- we happen to know initial value
result := 1;
FOR r IN
SELECT *
FROM mytable m
ORDER BY m.name, m.round
LOOP
IF (r.name, r.round) <> (name, round) THEN -- return result before round
RETURN NEXT;
name := r.name;
round := r.round;
result := 1;
END IF;
result := result * (1 - r.val/100);
END LOOP;
RETURN NEXT; -- return final result
END
$func$;
Call:
SELECT * FROM f_grp_prod();
Result:
name | round | result
-----+-------+---------------
A | 1 | 0.90430333812
A | 2 | 0.653458283632
Specific function as per question
CREATE OR REPLACE FUNCTION f_grp_prod(text)
RETURNS TABLE (name text
, result1 double precision
, result2 double precision)
LANGUAGE plpgsql STABLE AS
$func$
DECLARE
r mytable%ROWTYPE;
_round integer;
BEGIN
-- init vars
name := $1;
result2 := 1; -- abuse result2 as temp var for convenience
FOR r IN
SELECT *
FROM mytable m
WHERE m.name = name
ORDER BY m.round
LOOP
IF r.round <> _round THEN -- save result1 before 2nd round
result1 := result2;
result2 := 1;
END IF;
result2 := result2 * (1 - r.val/100);
_round := r.round;
END LOOP;
RETURN NEXT;
END
$func$;
Call:
SELECT * FROM f_grp_prod('A');
Result:
name | result1 | result2
-----+---------------+---------------
A | 0.90430333812 | 0.653458283632

I guess you are looking for an aggregate "product" function. You can create your own aggregate functions in Postgresql and Oracle.
CREATE TABLE mytable(name varchar(32), round int, position int, val decimal);
INSERT INTO mytable VALUES('A', 1, 1, 0.5);
INSERT INTO mytable VALUES('A', 1, 2, 3.4);
INSERT INTO mytable VALUES('A', 1, 3, 2.2);
INSERT INTO mytable VALUES('A', 1, 4, 3.8);
INSERT INTO mytable VALUES('A', 2, 1, 0.5);
INSERT INTO mytable VALUES('A', 2, 2, 32.3);
INSERT INTO mytable VALUES('A', 2, 3, 2.21);
INSERT INTO mytable VALUES('A', 2, 4, 0.8);
CREATE AGGREGATE product(double precision) (SFUNC=float8mul, STYPE=double precision, INITCOND=1);
SELECT name, round, product(1-val/100) AS result
FROM mytable
GROUP BY name, round;
name | round | result
------+-------+----------------
A | 2 | 0.653458283632
A | 1 | 0.90430333812
(2 rows)
See "User-Defined Aggregates" in the Postgresql doc. The example above I borrowed from
here. There are other stackoverflow responses that show other methods to do this.

Related

Dynamically fetch columns as JSON in oracle

I have to get some columns as is and some columns from a query as JSON document. The as is column names are known to me but rest are dynamic columns so there are not known beforehand.
Below is the query like
select col1,col2,col3,
sum(col4) as col4,
sum(col5) as col5
from my_table
group by col1,col2,col3;
here col4,col5 names are unknown to me as they are been fetched dynamically.
Suppost my_table data looks like
The expected result is like below
I tried
select JSON_OBJECT(*) from
(
select col1,col2,col3,
sum(col4) as col4,
sum(col5) as col5
from my_table
group by col1,col2,col3
);
But obviously it does not yield expected output.
I'm on 19c DB version 19.17
Any help or suggestion would be great help!
It's kinda hacky, but you could:
Use json_object(*) to convert the whole row to json
Pass the result of this json_transform*, which you can use to remove unwanted attributes
So you could do something like:
with rws as (
select mod ( level, 2 ) col1, mod ( level, 3 ) col2,
level col3, level col4
from dual
connect by level <= 10
), grps as (
select col1,col2,
sum(col3) as col3,
sum(col4) as col4
from rws
group by col1,col2
)
select col1,col2,
json_transform (
json_object(*),
remove '$.COL1',
remove '$.COL2'
) json_data
from grps;
COL1 COL2 JSON_DATA
---------- ---------- ------------------------------
1 1 {"COL3":8,"COL4":8}
0 2 {"COL3":10,"COL4":10}
1 0 {"COL3":12,"COL4":12}
0 1 {"COL3":14,"COL4":14}
1 2 {"COL3":5,"COL4":5}
0 0 {"COL3":6,"COL4":6}
json_transform is a 21c feature that's been backported to 19c in 19.10.
You may achieve this by using Polymorphic Table Function available since 18c.
Define the function that will project only specific columns and serialize others into JSON. An implementation is below.
PTF package (function implementation).
create package pkg_jsonify as
/*Package to implement PTF.
Functions below implement the API
described in the DBMS_TF package*/
function describe(
tab in out dbms_tf.table_t,
keep_cols in dbms_tf.columns_t
) return dbms_tf.describe_t
;
procedure fetch_rows;
end pkg_jsonify;
/
create package body pkg_jsonify as
function describe(
tab in out dbms_tf.table_t,
keep_cols in dbms_tf.columns_t
) return dbms_tf.describe_t
as
add_cols dbms_tf.columns_new_t;
new_col_cnt pls_integer := 0;
begin
for i in 1..tab.column.count loop
/*Initially remove column from the output*/
tab.column(i).pass_through := FALSE;
/*and keep it in the row processing (to be available for serialization*/
tab.column(i).for_read := TRUE;
for j in 1..keep_cols.count loop
/*If column is in a projection list, then remove it
from processing and pass it as is*/
if tab.column(i).description.name = keep_cols(j)
then
tab.column(i).pass_through := TRUE;
/*Skip column in the row processing (JSON serialization)*/
tab.column(i).for_read := FALSE;
end if;
end loop;
end loop;
/*Define new output column*/
add_cols := dbms_tf.columns_new_t(
1 => dbms_tf.column_metadata_t(
name => 'JSON_DOC_DATA',
type => dbms_tf.type_clob
)
);
/*Return the list of new cols*/
return dbms_tf.describe_t(
new_columns => add_cols
);
end;
procedure fetch_rows
/*Process rowset and serialize cols*/
as
rowset dbms_tf.row_set_t;
num_rows pls_integer;
new_col dbms_tf.tab_clob_t;
begin
/*Get rows*/
dbms_tf.get_row_set(
rowset => rowset,
row_count => num_rows
);
for rn in 1..num_rows loop
/*Calculate new column value in the same row*/
new_col(rn) := dbms_tf.row_to_char(
rowset => rowset,
rid => rn,
format => dbms_tf.FORMAT_JSON
);
end loop;
/*Put column to output*/
dbms_tf.put_col(
columnid => 1,
collection => new_col
);
end;
end pkg_jsonify;
/
PTF function definition based on the package.
create function f_cols_to_json(tab in table, cols in columns)
/*Function to serialize into JSON using PTF*/
return table pipelined
row polymorphic using pkg_jsonify;
/
Demo.
create table sample_tab
as
select
trunc(level/10) as id
, mod(level, 3) as id2
, level as val1
, level * level as val2
from dual
connect by level < 40
with prep as (
select
id
, id2
, sum(val1) as val1_sum
, max(val2) as val2_max
from sample_tab
group by
id
, id2
)
select *
from table(f_cols_to_json(prep, columns(id, id2)))
ID
ID2
JSON_DOC_DATA
0
1
{"VAL1_SUM":12, "VAL2_MAX":49}
0
2
{"VAL1_SUM":15, "VAL2_MAX":64}
0
0
{"VAL1_SUM":18, "VAL2_MAX":81}
1
1
{"VAL1_SUM":58, "VAL2_MAX":361}
1
2
{"VAL1_SUM":42, "VAL2_MAX":289}
1
0
{"VAL1_SUM":45, "VAL2_MAX":324}
2
2
{"VAL1_SUM":98, "VAL2_MAX":841}
2
0
{"VAL1_SUM":72, "VAL2_MAX":729}
2
1
{"VAL1_SUM":75, "VAL2_MAX":784}
3
0
{"VAL1_SUM":138, "VAL2_MAX":1521}
3
1
{"VAL1_SUM":102, "VAL2_MAX":1369}
3
2
{"VAL1_SUM":105, "VAL2_MAX":1444}
with prep as (
select
id
, id2
, sum(val1) as val1_sum
, max(val2) as val2_max
from sample_tab
group by
id
, id2
)
select *
from table(f_cols_to_json(prep, columns(id)))
ID
JSON_DOC_DATA
0
{"ID2":1, "VAL1_SUM":12, "VAL2_MAX":49}
0
{"ID2":2, "VAL1_SUM":15, "VAL2_MAX":64}
0
{"ID2":0, "VAL1_SUM":18, "VAL2_MAX":81}
1
{"ID2":1, "VAL1_SUM":58, "VAL2_MAX":361}
1
{"ID2":2, "VAL1_SUM":42, "VAL2_MAX":289}
1
{"ID2":0, "VAL1_SUM":45, "VAL2_MAX":324}
2
{"ID2":2, "VAL1_SUM":98, "VAL2_MAX":841}
2
{"ID2":0, "VAL1_SUM":72, "VAL2_MAX":729}
2
{"ID2":1, "VAL1_SUM":75, "VAL2_MAX":784}
3
{"ID2":0, "VAL1_SUM":138, "VAL2_MAX":1521}
3
{"ID2":1, "VAL1_SUM":102, "VAL2_MAX":1369}
3
{"ID2":2, "VAL1_SUM":105, "VAL2_MAX":1444}
fiddle

Convert JSON string in to separate fields

I have a table with two columns:
create table customerData (id bigint IDENTITY(1,1) NOT NULL, rawData varchar(max))
here the rawData will save the json format data in string, for example below will be the data in that column:
insert into customerData
values ('[{"customerName":"K C Nalina","attendance":"P","collectedAmount":"757","isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"1917889","totalDue":"757"},{"customerName":"Mahalakshmi","attendance":"P","collectedAmount":"881","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"430833","totalDue":"757"}]'),
('[{"customerName":"John","attendance":"P","collectedAmount":"700","isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"192222","totalDue":"788"},{"customerName":"weldon","attendance":"P","collectedAmount":"771","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"435874","totalDue":"757"}]')
Expected result :
I need these customerName, customerNumber, loanDisbProduct to be shown in separate fields for each rows.
Also to note the customer details inside rawData for each row will be more than two in many cases.
I don't know how to shred the data inside rawData column.
And I'm using SQL server 2012 and it doesn't support JSON data so I have to manipulate the string and get the field.
Thanks to Red-Gate blog post, first define a View as follow:(I will use this view to generate a new uniqueidentifier inside the function)
CREATE VIEW getNewID as SELECT NEWID() AS new_id
Then create a function as follow(This function is same as the one in Red-Gate blog post, but I have changed it a little a bit and include the identifier in it):
CREATE FUNCTION dbo.parseJSON( #JSON NVARCHAR(MAX))
RETURNS #hierarchy TABLE
(
Element_ID INT IDENTITY(1, 1) NOT NULL, /* internal surrogate primary key gives the order of parsing and the list order */
SequenceNo [int] NULL, /* the place in the sequence for the element */
Parent_ID INT null, /* if the element has a parent then it is in this column. The document is the ultimate parent, so you can get the structure from recursing from the document */
Object_ID INT null, /* each list or object has an object id. This ties all elements to a parent. Lists are treated as objects here */
Name NVARCHAR(2000) NULL, /* the Name of the object */
StringValue NVARCHAR(MAX) NOT NULL,/*the string representation of the value of the element. */
ValueType VARCHAR(10) NOT NULL, /* the declared type of the value represented as a string in StringValue*/
Identifier UNIQUEIDENTIFIER NOT NULL
)
AS
BEGIN
DECLARE
#FirstObject INT, --the index of the first open bracket found in the JSON string
#OpenDelimiter INT,--the index of the next open bracket found in the JSON string
#NextOpenDelimiter INT,--the index of subsequent open bracket found in the JSON string
#NextCloseDelimiter INT,--the index of subsequent close bracket found in the JSON string
#Type NVARCHAR(10),--whether it denotes an object or an array
#NextCloseDelimiterChar CHAR(1),--either a '}' or a ']'
#Contents NVARCHAR(MAX), --the unparsed contents of the bracketed expression
#Start INT, --index of the start of the token that you are parsing
#end INT,--index of the end of the token that you are parsing
#param INT,--the parameter at the end of the next Object/Array token
#EndOfName INT,--the index of the start of the parameter at end of Object/Array token
#token NVARCHAR(200),--either a string or object
#value NVARCHAR(MAX), -- the value as a string
#SequenceNo int, -- the sequence number within a list
#Name NVARCHAR(200), --the Name as a string
#Parent_ID INT,--the next parent ID to allocate
#lenJSON INT,--the current length of the JSON String
#characters NCHAR(36),--used to convert hex to decimal
#result BIGINT,--the value of the hex symbol being parsed
#index SMALLINT,--used for parsing the hex value
#Escape INT,--the index of the next escape character
#Identifier UNIQUEIDENTIFIER
DECLARE #Strings TABLE /* in this temporary table we keep all strings, even the Names of the elements, since they are 'escaped' in a different way, and may contain, unescaped, brackets denoting objects or lists. These are replaced in the JSON string by tokens representing the string */
(
String_ID INT IDENTITY(1, 1),
StringValue NVARCHAR(MAX)
)
SELECT--initialise the characters to convert hex to ascii
#characters='0123456789abcdefghijklmnopqrstuvwxyz',
#SequenceNo=0, --set the sequence no. to something sensible.
/* firstly we process all strings. This is done because [{} and ] aren't escaped in strings, which complicates an iterative parse. */
#Parent_ID=0,
#Identifier = (SELECT new_id FROM dbo.getNewID)
WHILE 1=1 --forever until there is nothing more to do
BEGIN
SELECT
#start=PATINDEX('%[^a-zA-Z]["]%', #json collate SQL_Latin1_General_CP850_Bin);--next delimited string
IF #start=0 BREAK --no more so drop through the WHILE loop
IF SUBSTRING(#json, #start+1, 1)='"'
BEGIN --Delimited Name
SET #start=#Start+1;
SET #end=PATINDEX('%[^\]["]%', RIGHT(#json, LEN(#json+'|')-#start) collate SQL_Latin1_General_CP850_Bin);
END
IF #end=0 --either the end or no end delimiter to last string
BEGIN-- check if ending with a double slash...
SET #end=PATINDEX('%[\][\]["]%', RIGHT(#json, LEN(#json+'|')-#start) collate SQL_Latin1_General_CP850_Bin);
IF #end=0 --we really have reached the end
BEGIN
BREAK --assume all tokens found
END
END
SELECT #token=SUBSTRING(#json, #start+1, #end-1)
--now put in the escaped control characters
SELECT #token=REPLACE(#token, FromString, ToString)
FROM
(SELECT '\b', CHAR(08)
UNION ALL SELECT '\f', CHAR(12)
UNION ALL SELECT '\n', CHAR(10)
UNION ALL SELECT '\r', CHAR(13)
UNION ALL SELECT '\t', CHAR(09)
UNION ALL SELECT '\"', '"'
UNION ALL SELECT '\/', '/'
) substitutions(FromString, ToString)
SELECT #token=Replace(#token, '\\', '\')
SELECT #result=0, #escape=1
--Begin to take out any hex escape codes
WHILE #escape>0
BEGIN
SELECT #index=0,
--find the next hex escape sequence
#escape=PATINDEX('%\x[0-9a-f][0-9a-f][0-9a-f][0-9a-f]%', #token collate SQL_Latin1_General_CP850_Bin)
IF #escape>0 --if there is one
BEGIN
WHILE #index<4 --there are always four digits to a \x sequence
BEGIN
SELECT --determine its value
#result=#result+POWER(16, #index)
*(CHARINDEX(SUBSTRING(#token, #escape+2+3-#index, 1),
#characters)-1), #index=#index+1 ;
END
-- and replace the hex sequence by its unicode value
SELECT #token=STUFF(#token, #escape, 6, NCHAR(#result))
END
END
--now store the string away
INSERT INTO #Strings (StringValue) SELECT #token
-- and replace the string with a token
SELECT #JSON=STUFF(#json, #start, #end+1,
'#string'+CONVERT(NCHAR(5), ##identity))
END
-- all strings are now removed. Now we find the first leaf.
WHILE 1=1 --forever until there is nothing more to do
BEGIN
SELECT #Parent_ID=#Parent_ID+1, #Identifier=(SELECT new_id FROM dbo.getNewID)
--find the first object or list by looking for the open bracket
SELECT #FirstObject=PATINDEX('%[{[[]%', #json collate SQL_Latin1_General_CP850_Bin)--object or array
IF #FirstObject = 0 BREAK
IF (SUBSTRING(#json, #FirstObject, 1)='{')
SELECT #NextCloseDelimiterChar='}', #type='object'
ELSE
SELECT #NextCloseDelimiterChar=']', #type='array'
SELECT #OpenDelimiter=#firstObject
WHILE 1=1 --find the innermost object or list...
BEGIN
SELECT
#lenJSON=LEN(#JSON+'|')-1
--find the matching close-delimiter proceeding after the open-delimiter
SELECT
#NextCloseDelimiter=CHARINDEX(#NextCloseDelimiterChar, #json,
#OpenDelimiter+1)
--is there an intervening open-delimiter of either type
SELECT #NextOpenDelimiter=PATINDEX('%[{[[]%',
RIGHT(#json, #lenJSON-#OpenDelimiter)collate SQL_Latin1_General_CP850_Bin)--object
IF #NextOpenDelimiter=0
BREAK
SELECT #NextOpenDelimiter=#NextOpenDelimiter+#OpenDelimiter
IF #NextCloseDelimiter<#NextOpenDelimiter
BREAK
IF SUBSTRING(#json, #NextOpenDelimiter, 1)='{'
SELECT #NextCloseDelimiterChar='}', #type='object'
ELSE
SELECT #NextCloseDelimiterChar=']', #type='array'
SELECT #OpenDelimiter=#NextOpenDelimiter
END
---and parse out the list or Name/value pairs
SELECT
#contents=SUBSTRING(#json, #OpenDelimiter+1,
#NextCloseDelimiter-#OpenDelimiter-1)
SELECT
#JSON=STUFF(#json, #OpenDelimiter,
#NextCloseDelimiter-#OpenDelimiter+1,
'#'+#type+CONVERT(NCHAR(5), #Parent_ID))
WHILE (PATINDEX('%[A-Za-z0-9#+.e]%', #contents collate SQL_Latin1_General_CP850_Bin))<>0
BEGIN
IF #Type='object' --it will be a 0-n list containing a string followed by a string, number,boolean, or null
BEGIN
SELECT
#SequenceNo=0,#end=CHARINDEX(':', ' '+#contents)--if there is anything, it will be a string-based Name.
SELECT #start=PATINDEX('%[^A-Za-z#][#]%', ' '+#contents collate SQL_Latin1_General_CP850_Bin)--AAAAAAAA
SELECT #token=RTrim(Substring(' '+#contents, #start+1, #End-#Start-1)),
#endofName=PATINDEX('%[0-9]%', #token collate SQL_Latin1_General_CP850_Bin),
#param=RIGHT(#token, LEN(#token)-#endofName+1)
SELECT
#token=LEFT(#token, #endofName-1),
#Contents=RIGHT(' '+#contents, LEN(' '+#contents+'|')-#end-1)
SELECT #Name=StringValue FROM #strings
WHERE string_id=#param --fetch the Name
END
ELSE
SELECT #Name=null,#SequenceNo=#SequenceNo+1
SELECT
#end=CHARINDEX(',', #contents)-- a string-token, object-token, list-token, number,boolean, or null
IF #end=0
--HR Engineering notation bugfix start
IF ISNUMERIC(#contents) = 1
SELECT #end = LEN(#contents) + 1
Else
--HR Engineering notation bugfix end
SELECT #end=PATINDEX('%[A-Za-z0-9#+.e][^A-Za-z0-9#+.e]%', #contents+' ' collate SQL_Latin1_General_CP850_Bin) + 1
SELECT
#start=PATINDEX('%[^A-Za-z0-9#+.e][A-Za-z0-9#+.e]%', ' '+#contents collate SQL_Latin1_General_CP850_Bin)
--select #start,#end, LEN(#contents+'|'), #contents
SELECT
#Value=RTRIM(SUBSTRING(#contents, #start, #End-#Start)),
#Contents=RIGHT(#contents+' ', LEN(#contents+'|')-#end)
IF SUBSTRING(#value, 1, 7)='#object'
INSERT INTO #hierarchy
(Name, SequenceNo, Parent_ID, StringValue, Object_ID, ValueType, Identifier)
SELECT #Name, #SequenceNo, #Parent_ID, SUBSTRING(#value, 8, 5),
SUBSTRING(#value, 8, 5), 'object' , #Identifier
ELSE
IF SUBSTRING(#value, 1, 6)='#array'
INSERT INTO #hierarchy
(Name, SequenceNo, Parent_ID, StringValue, Object_ID, ValueType, Identifier)
SELECT #Name, #SequenceNo, #Parent_ID, SUBSTRING(#value, 7, 5),
SUBSTRING(#value, 7, 5), 'array' , #Identifier
ELSE
IF SUBSTRING(#value, 1, 7)='#string'
INSERT INTO #hierarchy
(Name, SequenceNo, Parent_ID, StringValue, ValueType, Identifier)
SELECT #Name, #SequenceNo, #Parent_ID, StringValue, 'string', #Identifier
FROM #strings
WHERE string_id=SUBSTRING(#value, 8, 5)
ELSE
IF #value IN ('true', 'false')
INSERT INTO #hierarchy
(Name, SequenceNo, Parent_ID, StringValue, ValueType, Identifier)
SELECT #Name, #SequenceNo, #Parent_ID, #value, 'boolean', #Identifier
ELSE
IF #value='null'
INSERT INTO #hierarchy
(Name, SequenceNo, Parent_ID, StringValue, ValueType, Identifier)
SELECT #Name, #SequenceNo, #Parent_ID, #value, 'null', #Identifier
ELSE
IF PATINDEX('%[^0-9]%', #value collate SQL_Latin1_General_CP850_Bin)>0
INSERT INTO #hierarchy
(Name, SequenceNo, Parent_ID, StringValue, ValueType,Identifier)
SELECT #Name, #SequenceNo, #Parent_ID, #value, 'real', #Identifier
ELSE
INSERT INTO #hierarchy
(Name, SequenceNo, Parent_ID, StringValue, ValueType, Identifier)
SELECT #Name, #SequenceNo, #Parent_ID, #value, 'int', #Identifier
if #Contents=' ' Select #SequenceNo=0
END
END
INSERT INTO #hierarchy (Name, SequenceNo, Parent_ID, StringValue, Object_ID, ValueType, Identifier)
SELECT '-',1, NULL, '', #Parent_ID-1, #type, #Identifier
--
RETURN
END
Finally, If we have this table and data:
DECLARE #customerData TABLE (jsonValue NVARCHAR(MAX))
INSERT INTO #customerData
VALUES ('[{"customerName":"K C Nalina","attendance":"P","collectedAmount":"757","isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"1917889","totalDue":"757"},{"customerName":"Mahalakshmi","attendance":"P","collectedAmount":"881","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"430833","totalDue":"757"}]'),
('[{"customerName":"John","attendance":"P","collectedAmount":"700", "isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"192222","totalDue":"788"},{"customerName":"weldon","attendance":"P","collectedAmount":"771","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"435874","totalDue":"757"}]')
We can simply parse the JSON value as below:
;WITH jsonValue AS(
SELECT * FROM #customerData
CROSS APPLY(SELECT * FROM dbo.parseJSON(jsonvalue)) AS d
WHERE d.Name IN('customerName', 'customerNumber', 'loanDisbProduct')
)
,openResult AS(
SELECT i.Name, i.StringValue, i.Identifier FROM jsonValue AS i
)
SELECT
MAX(K.CustomerName) AS CustomerName,
MAX(K.CustomerNumber) AS CustomerNumber,
MAX(K.LoanDisbProduct) AS LoanDisbProduct
FROM (
SELECT
CASE WHEN openResult.Name='customerName' THEN openResult.StringValue ELSE NULL END AS CustomerName,
CASE WHEN openResult.Name='customerNumber' THEN openResult.StringValue ELSE NULL END AS CustomerNumber,
CASE WHEN openResult.Name='loanDisbProduct' THEN openResult.StringValue ELSE NULL END AS LoanDisbProduct,
openResult.Identifier
FROM openResult
) AS K
GROUP BY K.Identifier
And we will get the following output:
CustomerName | CustomerNumber | LoanDisbProduct
------------------------------------------------------
John | 192222 | null
Mahalakshmi | 430833 | Emergency Loan
K C Nalina | 1917889 | null
weldon | 435874 | Emergency Loan
If you do not know how many customers for each row, you shouldn't shred each customer to one field, at least a row pr customer.
Here is a start on shredding the data, I am using the dbo.STRING_SPLIT function from this page:
First I split by {} in the Json, then I split by ',', and then You ave the attribute name and value for each ID, with numbering of the customers in each row.
I could have split on ',' the same way as for '{...}' however I chose to use a function for this.
Everything is reliant on the same structure of the JSON. To do better parsing SQL server 2016+ would be recommended.
DROP TABLE IF EXISTS #customerData
create table #customerData (id bigint IDENTITY(1,1) NOT NULL, rawData varchar(max))
INSERT INTO #customerData
VALUES ('[{"customerName":"K C Nalina","attendance":"P","collectedAmount":"757","isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"1917889","totalDue":"757"},{"customerName":"Mahalakshmi","attendance":"P","collectedAmount":"881","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"430833","totalDue":"757"}]'),
('[{"customerName":"John","attendance":"P","collectedAmount":"700","isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"192222","totalDue":"788"},{"customerName":"weldon","attendance":"P","collectedAmount":"771","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"435874","totalDue":"757"}]')
;
WITH cte AS
(
SELECT id
, REPLACE(REPLACE(REPLACE(REPLACE(SUBSTRING(rawData, CHARINDEX('{', rawData), CHARINDEX('}', rawData) - CHARINDEX('{', rawData)), '{', ''), '[', ''), '}', ''), ']', '') person
, SUBSTRING(rawData, CHARINDEX('}', rawData) + 1, LEN(rawData)) personrest
, 1 nr
FROM #customerData
UNION ALL
SELECT id
, REPLACE(REPLACE(REPLACE(REPLACE(SUBSTRING(personrest, CHARINDEX('{', personrest), CHARINDEX('}', personrest) - CHARINDEX('{', personrest)), '{', ''), '[', ''), '}', ''), ']', '')
, SUBSTRING(personrest, CHARINDEX('}', personrest) + 1, LEN(personrest)) personrest
, nr + 1
FROM cte
WHERE CHARINDEX('}', personrest) > 0
AND CHARINDEX('{', personrest) > 0
)
SELECT id
, a.nr CustomerOrder
, LEFT([value], CHARINDEX(':', [value]) - 1)
, SUBSTRING([value], CHARINDEX(':', [value]) + 1, LEN([value]))
FROM cte a
CROSS APPLY (
SELECT *
FROM dbo.STRING_SPLIT(REPLACE(a.person, '"', ''), ',')
) b
The result is:
+─────+────────────────+──────────────────+─────────────────+
| id | CustomerOrder | Attribute | value |
+─────+────────────────+──────────────────+─────────────────+
| 1 | 1 | customerName | K C Nalina |
| 1 | 1 | attendance | P |
| 1 | 1 | collectedAmount | 757 |
| 1 | 1 | isOverdrafted | false |
| 1 | 1 | loanDisbProduct | null |
| 1 | 1 | paidBy | Y |
| 1 | 1 | customerNumber | 1917889 |
| 1 | 1 | totalDue | 757 |
| 2 | 1 | customerName | John |
| 2 | 1 | attendance | P |
| 2 | 1 | collectedAmount | 700 |
| 2 | 1 | isOverdrafted | false |
| 2 | 1 | loanDisbProduct | null |
| 2 | 1 | paidBy | Y |
| 2 | 1 | customerNumber | 192222 |
| 2 | 1 | totalDue | 788 |
| 2 | 2 | customerName | weldon |
| 2 | 2 | attendance | P |
| 2 | 2 | collectedAmount | 771 |
| 2 | 2 | isOverdrafted | false |
| 2 | 2 | loanDisbProduct | Emergency Loan |
| 2 | 2 | paidBy | Y |
| 2 | 2 | customerNumber | 435874 |
| 2 | 2 | totalDue | 757 |
| 1 | 2 | customerName | Mahalakshmi |
| 1 | 2 | attendance | P |
| 1 | 2 | collectedAmount | 881 |
| 1 | 2 | isOverdrafted | false |
| 1 | 2 | loanDisbProduct | Emergency Loan |
| 1 | 2 | paidBy | Y |
| 1 | 2 | customerNumber | 430833 |
| 1 | 2 | totalDue | 757 |
+─────+────────────────+──────────────────+─────────────────+
Best was to upgrade to v2016+. With JSON support this was easy...
On v2012 you have to hack around. It might be a better choice to use another tool for this. But, if you have to stick to TSQL, I would try to transform the JSON to attribute centered XML like here:
DECLARE #customerData TABLE (id bigint IDENTITY(1,1) NOT NULL, rawData varchar(max));
insert into #customerData
values ('[{"customerName":"K C Nalina","attendance":"P","collectedAmount":"757","isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"1917889","totalDue":"757"},{"customerName":"Mahalakshmi","attendance":"P","collectedAmount":"881","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"430833","totalDue":"757"}]'),
('[{"customerName":"John","attendance":"P","collectedAmount":"700","isOverdrafted":false,"loanDisbProduct":null,"paidBy":"Y","customerNumber":"192222","totalDue":"788"},{"customerName":"weldon","attendance":"P","collectedAmount":"771","isOverdrafted":false,"loanDisbProduct":"Emergency Loan","paidBy":"Y","customerNumber":"435874","totalDue":"757"}]')
--the query
SELECT cd.id
,B.*
FROM #customerData cd
CROSS APPLY(SELECT REPLACE(REPLACE(REPLACE(cd.rawData,'false','"0"'),'true','"1"'),'null','"#NULL"')) A(JustStringValues)
CROSS APPLY(SELECT CAST(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(JustStringValues,'[',''),']',''),'},{"',' /><x '),'{"','<x '),'}',' />'),'","','" '),'":"','="') AS XML)) B(SinlgeRow)
--the result
<x customerName="K C Nalina" attendance="P" collectedAmount="757" isOverdrafted="0" loanDisbProduct="#NULL" paidBy="Y" customerNumber="1917889" totalDue="757" /x>
<x customerName="Mahalakshmi" attendance="P" collectedAmount="881" isOverdrafted="0" loanDisbProduct="Emergency Loan" paidBy="Y" customerNumber="430833" totalDue="757" /x>
<x customerName="John" attendance="P" collectedAmount="700" isOverdrafted="0" loanDisbProduct="#NULL" paidBy="Y" customerNumber="192222" totalDue="788" /x>
<x customerName="weldon" attendance="P" collectedAmount="771" isOverdrafted="0" loanDisbProduct="Emergency Loan" paidBy="Y" customerNumber="435874" totalDue="757" /x>
The idea in short:
We replace the non-quoted values (false, true, null) with a quoted place holder
We use various replacements to get the attribute centered XML
Use this query to get the values
SELECT cd.id
,OneCustomer.value('#customerName','nvarchar(max)') AS CustomerName
,OneCustomer.value('#attendance','nvarchar(max)') AS Attendance
--more attributes
FROM #customerData cd
CROSS APPLY(SELECT REPLACE(REPLACE(REPLACE(cd.rawData,'false','"0"'),'true','"1"'),'null','"#NULL"')) A(JustStringValues)
CROSS APPLY(SELECT CAST(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(JustStringValues,'[',''),']',''),'},{"',' /><x '),'{"','<x '),'}',' />'),'","','" '),'":"','="') AS XML)) B(SinlgeRow)
CROSS APPLY B.SinlgeRow.nodes('/x') AS C(OneCustomer);

How to show positions of '1' within a long number mysql

I know I can find the first position of 1 with my number by using the following:
SELECT POSITION("1" IN "0000100001000001");
How would I find all the positions of 1 to return 5;10;16
Your string appears to be a 16-bit number represented in base-2.
I set a user variable to your example string.
set #bin = '0000100001000001';
We can use CONV() to convert it to a base-10 number instead of base-2. This allows the integer value to be used when we use it in numeric expressions.
mysql> select conv(#bin, 2, 10);
+-------------------+
| conv(#bin, 2, 10) |
+-------------------+
| 2113 |
+-------------------+
Then we can test for a particular bit set in this number using the & bitwise-and operator.
mysql> select conv(#bin, 2, 10) & 64;
+------------------------+
| conv(#bin, 2, 10) & 64 |
+------------------------+
| 64 |
+------------------------+
We can test all the bits of the integer value. If a given bit is set, then substitute the "position," as you call it, for that bit (counting from left to right, which is the opposite of the traditional bit positions).
If the bit is not set, then default to NULL. Then concatenate these together using CONCAT_WS(), which ignores NULLs.
select concat_ws(';',
case conv(#bin,2,10)&32768 when 32768 then 1 end,
case conv(#bin,2,10)&16384 when 16384 then 2 end,
case conv(#bin,2,10)&8192 when 8192 then 3 end,
case conv(#bin,2,10)&4096 when 4096 then 4 end,
case conv(#bin,2,10)&2048 when 2048 then 5 end,
case conv(#bin,2,10)&1024 when 1024 then 6 end,
case conv(#bin,2,10)&512 when 512 then 7 end,
case conv(#bin,2,10)&256 when 256 then 8 end,
case conv(#bin,2,10)&128 when 128 then 9 end,
case conv(#bin,2,10)&64 when 64 then 10 end,
case conv(#bin,2,10)&32 when 32 then 11 end,
case conv(#bin,2,10)&16 when 16 then 12 end,
case conv(#bin,2,10)&8 when 8 then 13 end,
case conv(#bin,2,10)&4 when 4 then 14 end,
case conv(#bin,2,10)&2 when 2 then 15 end,
case conv(#bin,2,10)&1 when 1 then 16 end) as bits_set;
Output:
+----------+
| bits_set |
+----------+
| 5;10;16 |
+----------+
There is no such functionality built in. You can create your own function for this.
delimiter $$
create function f_position_multiple(
in_f char(1),
in_str text
)
returns text
begin
declare v_delim char(1);
declare v_loc int;
declare v_ret text;
set v_ret = '';
set v_delim = '';
set v_loc = 0;
set v_loc = locate(in_f, in_str, v_loc+1);
while(v_loc>0) do
set v_ret = concat(v_ret, v_delim, v_loc);
set v_delim = ';';
set v_loc = locate(in_f, in_str, v_loc+1);
end while;
return v_ret;
end
$$
And then you can use:
select f_position_multiple('1', '1001001')
A solution for MySql 8.0+ with a recursive CTE:
set #n = '0000100001000001';
with recursive cte as (
select 0 pos, ' ' bit
union all
select pos + 1, substring(#n, pos + 1, 1)
from cte
where pos < length(#n)
)
select group_concat(pos order by pos separator ';') result
from cte
where bit = '1'
See the demo.
Result:
| result |
| ------- |
| 5;10;16 |

MySQL Select multiple columns group by sorted columns values

I have this table columns structure:
id - n1 - n2 - n3
And here it is with some dummy data:
id - n1 - n2 - n3
1 - 3 - 2 - 1
2 - 6 - 5 - 7
3 - 2 - 3 - 1
4 - 1 - 6 - 5
5 - 5 - 6 - 7
6 - 3 - 5 - 6
And the idea is to Select and count each unique distinct group of n1, n2 and n3 in sequence.
So, for example, we could get this result:
total - n1s - n2s - n3s
2 - 1 - 2 - 3
2 - 5 - 6 - 7
1 - 1 - 5 - 6
1 - 3 - 5 - 6
Can you help me set the state to achieve that??
I am trying to attempt that without multiple selects and PHP array sorting...
Thanks.
Consider the following - a normalised dataset...
DROP TABLE IF EXISTS my_table;
CREATE TABLE my_table
(id INT NOT NULL
,n INT NOT NULL
,val INT NOT NULL
,PRIMARY KEY(id,n)
);
INSERT INTO my_table VALUES
(1, 1, 3),
(1, 2, 2),
(1, 3, 1),
(2, 1, 6),
(2, 2, 5),
(2, 3, 7),
(3, 1, 2),
(3, 2, 3),
(3, 3, 1),
(4, 1, 1),
(4, 2, 6),
(4, 3, 5),
(5, 1, 5),
(5, 2, 6),
(5, 3, 7),
(6, 1, 3),
(6, 2, 5),
(6, 3, 6);
Here's a quick (to write) and dirty solution. Faster / more elegant solutions are available...
SELECT vals
, COUNT(*) total
FROM
( SELECT id
, GROUP_CONCAT(val ORDER BY val) vals
FROM my_table
GROUP
BY id
) x
GROUP
BY vals;
+-------+-------+
| vals | total |
+-------+-------+
| 1,2,3 | 2 |
| 1,5,6 | 1 |
| 3,5,6 | 1 |
| 5,6,7 | 2 |
+-------+-------+
We just need expressions to "sort" the values in columns n1, n2 and n3. If we have that, then we can do a simple GROUP BY and COUNT.
SELECT COUNT(1) AS total
, IF(t.n1<=t.n2,IF(t.n1<=t.n3,t.n1,t.n3),IF(t.n2<=t.n3,t.n2,t.n3)) AS n1s
, IF(t.n1<=t.n2,IF(t.n2<=t.n3,t.n2,IF(t.n1<=t.n3,t.n3,t.n1)),IF(t.n1<=t.n3,t.n1,IF(t.n2<=t.n3,t.n3,t.n2 ))) AS n2s
, IF(t.n1<=t.n2,IF(t.n2<=t.n3,t.n3,t.n2),IF(t.n1<=t.n3,t.n3,t.n1)) AS n3s
FROM this_table_column_structure t
GROUP BY n1s,n2s,n3s
ORDER BY total DESC, n1s, n2s, n3s
will return
total n1s n2s n3s
----- ---- ---- ----
2 1 2 3
2 5 6 7
1 1 5 6
1 3 5 6
As a first approach (if time permits), you should really consider normalizing your table, as suggested in #Strawberry's answer
However, a second approach allowing any number of columns (although inefficient due to String operations and Bubble Sorting) is possible, utilizing User Defined Functions.
We basically need to create a function, which can sort the values inside a comma separated string. I found a working function, which can do the sorting. Reproducing code from here:
-- sort comma separated substrings with unoptimized bubble sort
DROP FUNCTION IF EXISTS sortString;
DELIMITER |
CREATE FUNCTION sortString(inString TEXT) RETURNS TEXT
BEGIN
DECLARE delim CHAR(1) DEFAULT ','; -- delimiter
DECLARE strings INT DEFAULT 0; -- number of substrings
DECLARE forward INT DEFAULT 1; -- index for traverse forward thru substrings
DECLARE backward INT; -- index for traverse backward thru substrings, position in calc. substrings
DECLARE remain TEXT; -- work area for calc. no of substrings
-- swap areas TEXT for string compare, INT for numeric compare
DECLARE swap1 TEXT; -- left substring to swap
DECLARE swap2 TEXT; -- right substring to swap
SET remain = inString;
SET backward = LOCATE(delim, remain);
WHILE backward != 0 DO
SET strings = strings + 1;
SET backward = LOCATE(delim, remain);
SET remain = SUBSTRING(remain, backward+1);
END WHILE;
IF strings < 2 THEN RETURN inString; END IF;
REPEAT
SET backward = strings;
REPEAT
SET swap1 = SUBSTRING_INDEX(SUBSTRING_INDEX(inString,delim,backward-1),delim,-1);
SET swap2 = SUBSTRING_INDEX(SUBSTRING_INDEX(inString,delim,backward),delim,-1);
IF swap1 > swap2 THEN
SET inString = TRIM(BOTH delim FROM CONCAT_WS(delim
,SUBSTRING_INDEX(inString,delim,backward-2)
,swap2,swap1
,SUBSTRING_INDEX(inString,delim,(backward-strings))));
END IF;
SET backward = backward - 1;
UNTIL backward < 2 END REPEAT;
SET forward = forward +1;
UNTIL forward + 1 > strings
END REPEAT;
RETURN inString;
END |
DELIMITER ;
You will need to run this code on your MySQL server, so that this function is available within a query, just like native built-in MySQL functions. Now, the querying part becomes simple. All you need to do is Concat_ws() all the number columns using comma. And, then apply sortString() function on the concatenated string. Eventually, use the "ordered" string in Group By clause, to get the desired result.
Try:
SELECT sortString(CONCAT_WS(',', n1, n2, n3)) AS n_sequence -- add more columns here
COUNT(id) AS total
FROM your_table
GROUP BY n_sequence
ORDER BY total DESC
Now I suggest that you can use your application code to change comma separated n_sequence back to tabular column display.

How to import a text file with no delimiters with 2 lines representing a case

I have several text files that I need to import into MySQL, but they don't have any delimiters, and 3 lines in the text file represent one record.
When I try to import it everything goes into one column. Please see an example below
00003461020000001ACH1 00000000 00000000000 00000000 000000005011025708084 0 00 00 000000000000000000000 00000000241523551MA00
You need a helper table first.
CREATE TABLE tmpHelperTable(
your_data varchar(255),
a int,
b int
);
Then you need two user defined variables while loading your data.
SET #va = 0;
SET #vb = 0;
LOAD DATA INFILE 'your_data_file.csv'
INTO tmpHelperTable
LINES TERMINATED BY '\n'
(your_data, a, b)
SET a = #va := IF(#va = 3, 1, #va + 1),
b = IF(#va % 3 = 0, #vb := #vb + 1, #vb);
This line
SET a = #va := IF(#va = 3, 1, #va + 1),
is just an incrementing value, that resets when it reaches 3 (or whatever many lines determine one case).
The line
b = IF(#va = 1, #vb := #vb + 1, #vb);
just increments its value every time the previous variable got reset. We need this so we can group by it. Then you have a table like this:
your_data | a | b
xxxxxx 1 1
yyyyyy 2 1
zzzzzz 3 1
aaaaaa 1 2
bbbbbb 2 2
cccccc 3 2
dddddd 1 3
...
Then all you have to do is to pivot the table into your final table.
CREATE TABLE final_table(
id int,
data_1 varchar(255),
data_2 varchar(255),
data_3 varchar(255)
);
INSERT INTO final_table
SELECT
b,
MAX(IF(a = 1, your_data, NULL)),
MAX(IF(a = 2, your_data, NULL)),
MAX(IF(a = 3, your_data, NULL)),
FROM
tmpHelperTable
GROUP BY b;