we are trying to use PL/JSON package for our programs but it looks like that json.to_clob method has a bug. If you try this code:
-- json to clob bug in pl/json package
-- if json value is single character then that json value will not be shown in clob
declare
l_Data_json_clob clob;
l_Data_json json;
begin
l_Data_json := json();
dbms_lob.createtemporary(l_Data_json_clob, true); -- create CLOB where result will be stored
l_Data_json.put('RESULT', '1');--put one character in json value
l_Data_json.to_clob(l_Data_json_clob);--put json result to clob
dbms_output.put_line(dbms_lob.substr( l_Data_json_clob, 255, 1));--not correct json - json value is empty
l_Data_json.put('RESULT', '22');--put two characters in json value
l_Data_json.to_clob(l_Data_json_clob);--put json result to clob
dbms_output.put_line(dbms_lob.substr( l_Data_json_clob, 255, 1));--correct json - more then one character
l_Data_json.put('RESULT', 1);--put one number in json value
l_Data_json.to_clob(l_Data_json_clob);--put json result to clob
dbms_output.put_line(dbms_lob.substr( l_Data_json_clob, 255, 1));--again correct json but with number instead of string
end;
You will see that for some reason first json value is empty. We noticed that it is happening only if json value is single character. Is there any workaround for this behavior?
Thank you
After some debugging I found that the bug is in JSON_PRINTER.ppString. The correct code should be:
procedure ppString(elem json_value, buf in out nocopy clob, buf_str in out nocopy varchar2) is
offset number := 1;
v_str varchar(5000 char);
amount number := 5000; /*chunk size for use in escapeString. Maximum escaped unicode string size for chunk may be 6 one-byte chars * 5000 chunk size in multi-byte chars = 30000 1-byte chars. Maximum value may be 32767 1-byte chars */
begin
add_to_clob(buf, buf_str, case when elem.num = 1 then '"' else '/**/' end);
if(elem.extended_str is not null) then /*clob implementation*/
while(offset <= dbms_lob.getlength(elem.extended_str)) loop
dbms_lob.read(elem.extended_str, amount, offset, v_str);
if(elem.num = 1) then
add_to_clob(buf, buf_str, escapeString(v_str));
else
add_to_clob(buf, buf_str, v_str);
end if;
offset := offset + amount;
end loop;
else
if(elem.num = 1) then
while(offset**<=**length(elem.str)) loop
v_str:=substr(elem.str, offset, amount);
add_to_clob(buf, buf_str, escapeString(v_str));
offset := offset + amount;
end loop;
else
add_to_clob(buf, buf_str, elem.str);
end if;
end if;
add_to_clob(buf, buf_str, case when elem.num = 1 then '"' else '/**/' end || newline_char);
end;
Related
I am struggling to parse a JSON having more than 32767 keys
The format of json is shown below.
The idea is to fetch all the keys and value into a temp table , but due to length limitation unable to do so
{"1335":"345435sd","8989SD":"jddk8","dDDSF","87868658"......}
declare
j JSON_OBJECT_T;
i NUMBER;
k JSON_KEY_LIST;
arr JSON_ARRAY_T;
v_key varchar2(2000);
v_value varchar2(2000);
CURSOR c_json IS
select treat(col_clob as json) myJsonCol from t_clob; -- the data is stored as clob and it need to be
begin
FOR rec IN c_json
LOOP
j := JSON_OBJECT_T.parse(rec.myJsonCol);
k := j.get_keys;
FOR i in 1..k.COUNT
LOOP
dbms_output.put_line(k(i) || ' ' || j.get_String(k(i)));
v_key :=k(i);
v_value :=j.get_String(k(i));
-- insert into temp(c1,c2) values(v_key,v_value);
END LOOP;
END LOOP;
END;
/
::Oracle Error: ORA-40684: maximum number of key names exceeded::
Reference - https://docs.oracle.com/en/database/oracle/oracle-database/12.2/adjsn/oracle-json-restrictions.html#GUID-1DB81125-54A7-4CB6-864B-78E0E7E407C9
PL/SQL getter method JSON_OBJECT_T.get_keys() returns at most 32767 field names for a given JSON object. An error is raised if it is applied to an object with more than 32767 fields.
I'm trying to extract a very long string into clob from json_object_t and got some weird database behaviour (12.2c) with json_object_t.get_clob(key) method.
There is a sample code than does following:
DECLARE
l_data CLOB := '{"text": "very long string about 1M chars"}';
l_json json_object_t;
l_text CLOB := EMPTY_CLOB();
BEGIN
l_json := json_object_t.parse(l_data);
l_text := l_json.get_clob('text');
dbms_output.put_line('got ' || dbms_lob.getlength(l_text) || ' chars');
END;
When string length in a 'text' key is less than 32k chars, get_clob method works just fine and shows appropriate result, but with longer strings it produces an empty clob with zero length, just like get_string, but without 'character string buffer too small' exception.
I've tried to get same data via json_table query, but it cannot extract data to clob column at all, only varchar/number is allowed.
Is that a bug or am I doing something wrong? Is there any other ways to extract long strings from JSON keys?
I work with Oracle Database JSON Store group and would be happy to assist you with this issue you're facing. Could you try the alternate get_Clob procedure instead of this function and tell us what the behavior is?
Signature:
MEMBER PROCEDURE get_Clob(key VARCHAR2, c IN OUT CLOB)
Please try this:
DECLARE
content_json CLOB := '{"value":"';
content_json_end CLOB := '"}';
content_tmp CLOB := 'ab';
l_json json_object_t;
l_text CLOB := EMPTY_CLOB();
tmp clob;
BEGIN
-- 13 gives 16K
-- 14 gives 32K
FOR count IN 1 .. 14
loop
dbms_lob.append(content_tmp, content_tmp); -- a bad append for now
END loop;
dbms_lob.append(content_json, content_tmp);
dbms_lob.append(content_json, content_json_end);
l_json := json_object_t.parse(content_json);
l_json.get_clob('value', l_text); -- !!! TRY THIS PROC get_Clob
--l_text := l_json.get_clob('value');
dbms_output.put_line('Lob size in Kb: ');
dbms_output.put_line(dbms_lob.getLength(l_text) / 1024);
END;
/
Looking forward to your findings..
This works as well. Instead using the the get_clob method, use c:
DECLARE
CURSOR crsrJSON IS
SELECT
json_object( 'employee_id' VALUE employee_id,
'first_name' VALUE first_name,
'last_name' VALUE last_name,
'email' VALUE email,
'phone_number' VALUE phone_number,
'hire_date' VALUE to_char(hire_date,'MM/DD/YYYY'),
'job_id' VALUE job_id,
'salary' VALUE nvl(salary,0),
'commission_pct' VALUE nvl(commission_pct,0),
'manager_id' VALUE NVL(manager_id,0),
'department_id' VALUE NVL(department_id,0),
'department_name' VALUE (select department_name from departments x where x.department_id = hr.department_id),
'job_title' VALUE (select job_title from jobs x where x.job_id = hr.job_id)) emp_data
FROM
employees hr;
js_array JSON_ARRAY_T := new JSON_ARRAY_T;
json_obj JSON_OBJECT_T := JSON_OBJECT_T();
json_clob CLOB := EMPTY_CLOB();
BEGIN
FOR data_rec IN crsrJSON LOOP
js_array.append(JSON_ELEMENT_T.parse(data_rec.emp_data));
END LOOP;
json_obj.put('data',js_array);
IF json_obj.has('data') THEN
json_clob := json_obj.to_clob;
DBMS_OUTPUT.PUT_LINE(json_clob);
ELSE
DBMS_OUTPUT.PUT_LINE('Nope');
END IF;
END;
with data as
( select
xmlelement(e,regexp_replace('{"name":"'||colname||'"}', '[[:cntrl:]]', ''),',') col1
from tblname
)
select
rtrim(replace(replace(replace(xmlagg(col1).getclobval(),'&'||'quot;','"'),'<E>',''),'</E>',''),',')
as very_long_json
from data;
I'm migrating the database engine an application from MySql to SAP HANA.
I found a little trouble. I have a query like this:
Select SUBSTRING_INDEX(id, "-", -2) as prod_ref From products;
I don't know how to "translate" the function substring_index, because the initial part of the id has a variable length.
Thanks.
This can be done using a regex:
select substr_regexpr( '.*-([^-]*-[^-]*)$' in 'varia-ble---part-part1-part2' group 1) from dummy;
select substr_regexpr( '.*-([^-]*-[^-]*)$' in 'variable-part-part1-part2' group 1) from dummy;
According to the HANA 2.0 SP0 doc you could use locate with a negative offset (and then using right()), but this does not work on my system ("...feature isn't supported...")
If you execute such queries on a regular basis on lots of records I would recommend extracting the part you are interested in during ETL into a separate field. Or, alternatively fill a separate field using " GENERATED ALWAYS AS...".
I have seen it more than once, that people calculate a field like this in complex SQL queries or complex CalcViews, and then wonder why performance is bad when selecting 100 million records and filtering on the calculated field etc... Performance is usually no problem when you have aggregated your intermediate result set to a reasonable size and then apply "expensive" functions.
I don't think there is any direct function like SUBSTRING_INDEX in SAP HANA. But you have a work around alternative by creating a function to pass the input string and delimiter.
And I am assuming that -2 in SUBSTRING_INDEX and providing the solution
Reverse the string and get the position of the second delimiter, '-' in your case, into "obtainedPosition"
Now subtract that "obtainedPosition" from the length of the string.
obtainedPosition = LENGTH(id) - obtainedPosition
Using that value in the inbuilt substring function you can get the required string and return it from the function.
SELECT SCHEMA.FN_SUBSTRING_INDEX(id,obtainedPosition) INTO ReturnValue FROM DUMMY;
CREATE FUNCTION FN_SUBSTRING_INDEX
(
id VARCHAR(500),
delim VARCHAR(2)
)
RETURNS SplitString VARCHAR(500)
LANGUAGE SQLSCRIPT AS
BEGIN
DECLARE reversedString VARCHAR(500);
DECLARE charString VARCHAR(2);
DECLARE i INT := LENGTH(:id);
DECLARE len INT := LENGTH(:id);
DECLARE obtainedPosition INT := 0;
DECLARE flag INT := 0;
reversedString := '';
--loop to reverse the inputstring
WHILE :i > 0
DO
reversedString = CONCAT(:reversedString, SUBSTRING(:id,:i,1));
i := :i - 1;
END WHILE;
--loop to get the second delimiter position
i := 1;
WHILE :i <= :leng
DO
charString := '';
charString := SUBSTRING(:reversedString,i,1);
IF((:charString = :delim ) AND (:flag < 2)) THEN
BEGIN
obtainedPosition := :i;
flag := :flag + 1;
END;
END IF;
i := :i + 1;
END WHILE;
--IF condition to check if at least 2 delimiters are available, else print complete string
IF(flag = 2) THEN
obtainedPosition := :len - :obtainedPosition + 2; --2 is added to avoid the character at that position and '-' from printing
ELSE
obtainedPosition := 1;
END IF;
--SplitString contains the string's splitted return value
SELECT SUBSTRING(:id,:obtainedPosition) INTO SplitString FROM DUMMY;
END;
The above function is modified from http://www.kodyaz.com/sap-abap/sqlscript-reverse-string-function-in-sap-hana.aspx
For string functions in SAP HANA refer to this: http://www.sapstudent.com/hana/sql-string-functions-in-sap-hana/3
You can use anonymous block in SAP HANA to call and check the function
DO
BEGIN
DECLARE id VARCHAR(500) := 'Test-sam-ple-func';
DECLARE delim VARCHAR(2) := '-';
SELECT SCHEMA.FN_SUBSTRING_INDEX(id,delim) AS "SplitStringIndex" FROM DUMMY;
END;
I would be glad to know reason for a downvote. :)
For Oracle 11g doesn't support json as you know, i am creating json data using CLOB due to varchar2 (32767) size but in this time i am getting ORA-06502: PL/SQL: numeric or value error. Meanwhile, the data size is 68075. Why i am getting the error despite of Clob is support 4 gb data?
procedure course
(
p varchar2 default null
)
as
cursor cr_course(cp_param varchar2)
is
select
m.code,
m.title
from t_course m
where
and type = cp_param;
jobject clob;
jitem varchar2(200);
begin
dbms_lob.createtemporary(jobject, false);
for n in cr_course(p) loop
jitem := '{"key":"' || n.code || '", "value":"'|| n.title || '"},';
dbms_lob.append(jobject, jitem);
end loop;
htp.p(substr(jobject, 0, (length(jobject)-1)));
exception when others then
dbms_lob.freetemporary(jobject);
htp.p(sqlerrm);
end;
I think the error is because of htp.p(substr(jobject, 0, (length(jobject)-1)));
I don't think substr works on clob type. You will have to convert it to VARCHAR2 datatype to use substr on it.
I have a set of characters that are defined as valid characters. Let me define the valid string by the following regexp:
^[a-zA-Z0-9\ .-_]+$
(alphanumeric, space, dot, dash and underscore)
The question is that given a column containing a lot of invalid characters, how I can run an update to convert each invalid character to one space? And then possibly convert consequent spaces to one space?
I cannot run several replace commands because there are a lot of possible invalid characters. So I expect a regexp solution.
Currently, I am doing the task in Java (after exporting the table to tsv format). But I want a MySQL approach.
If your MySQL version supports it, create a function:
DELIMITER $$
CREATE FUNCTION my_func_1 (str TEXT)
RETURNS TEXT
BEGIN
DECLARE ret TEXT DEFAULT '';
DECLARE chr TEXT DEFAULT '';
DECLARE i INT DEFAULT 1;
WHILE i < (LENGTH(str) + 1) DO
SET chr = SUBSTRING(str, i, 1);
IF chr REGEXP '[-a-zA-Z0-9\\_.]'
THEN SET ret = CONCAT(ret, chr);
ELSE
SET ret = CONCAT(ret, ' ');
END IF;
SET i = i + 1;
END WHILE;
WHILE ret LIKE '% %' DO
SET ret = REPLACE(ret, ' ', ' ');
END WHILE;
RETURN TRIM(ret);
END;
$$
DELIMITER ;
Test it a bit:
SELECT my_func_1('$a-B\\?!=01._%'); > a-B\ 01._
and update with SET col = my_func_1(col)
If not needed anymore:
DROP FUNCTION IF EXISTS my_func_1;
Also I changed your regex a bit as - indicates a range, if between characters in a class or is .-_ intended? Then modify the pattern.