I get a json string drom my network reply it looks like
{\"topic\":\"ddd\",\"payload\":\"{\\\"eeee\\\":\\\"eee\\\",\\\"thingUID\\\":\\\"eee\\\",\\\"ee\\\":\\\"eee\\\",\\\"serialNumber\\\":\\\"eee\\\",\\\"epoch\\\":\\\"eeee\\\",\\\"eee\\\":\\\"eee\\\",\\\"eee\\\":\\\"eee\\\"}\",\"type\":\"eee\"}\n\n
how can i remove the best and fastest way all these backslahes "" in QT???
I do it like this
QByteArray data = _sseReply->readAll();
data = data.replace("\\\"", "\"");
data = data.replace("}\"", "}");
data = data.replace("\"{", "{");
any cooler solution available??
Just use QJsonDocument::fromJson(data).object() and don't need remove backslashes.
Related
I'm trying to convert a query from Redshift to Snowflake SQL.
The Redshift query looks like this:
SELECT
cr.creatives as creatives
, JSON_ARRAY_LENGTH(cr.creatives) as creatives_length
, JSON_EXTRACT_PATH_TEXT(JSON_EXTRACT_ARRAY_ELEMENT_TEXT (cr.creatives,0),'previewUrl') as preview_url
FROM campaign_revisions cr
The Snowflake query looks like this:
SELECT
cr.creatives as creatives
, ARRAY_SIZE(TO_ARRAY(ARRAY_CONSTRUCT(cr.creatives))) as creatives_length
, PARSE_JSON(PARSE_JSON(cr.creatives)[0]):previewUrl as preview_url
FROM campaign_revisions cr
It seems like JSON_EXTRACT_PATH_TEXT isn't converted correctly, as the Snowflake query results in error:
Error parsing JSON: more than one document in the input
cr.creatives is formatted like this:
"[{""previewUrl"":""https://someurl.com/preview1.png"",""device"":""desktop"",""splitId"":null,""splitType"":null},{""previewUrl"":""https://someurl.com/preview2.png"",""device"":""mobile"",""splitId"":null,""splitType"":null}]"
It seems to me that you are not working with valid JSON data inside Snowflake.
Please review your file format used for the copy into command.
If you open the "JSON" text provided in a text editor , note that the information is not parsed or formatted as JSON because of the quoting you have. Once your issue with double quotes / escaped quotes is handled, you should be able to make good progress
Proper JSON on Left || Original Data on Right
If you are not inclined to reload your data, see if you can create a Javascript User Defined Function to remove the quotes from your string, then you can use Snowflake to process the variant column.
The following code is working POJO that can be used to remove the doublequotes for you.
var textOriginal = '[{""previewUrl"":""https://someurl.com/preview1.png"",""device"":""desktop"",""splitId"":null,""splitType"":null},{""previewUrl"":""https://someurl.com/preview2.png"",""device"":""mobile"",""splitId"":null,""splitType"":null}]';
function parseText(input){
var a = input.replaceAll('""','\"');
a = JSON.parse(a);
return a;
}
x = parseText(textOriginal);
console.log(x);
For anyone else seeing this double double quote issue in JSON fields coming from CSV files in a Snowflake external stage (slightly different issue than the original question posted):
The issue is likely that you need to use the FIELD_OPTIONALLY_ENCLOSED_BY setting. Specifically, FIELD_OPTIONALLY_ENCLOSED_BY = '"' when setting up your fileformat.
(docs)
Example of creating such a file format:
create or replace file format mydb.myschema.my_tsv_file_format
type = CSV
field_delimiter = '\t'
FIELD_OPTIONALLY_ENCLOSED_BY = '"';
And example of querying from a stage using this file format:
select
$1 field_one
$2 field_two
-- ...and so on
from '#my_s3_stage/path/to/file/my_tab_separated_file.csv' (file_format => 'my_tsv_file_format')
I have a table I need to handle various characters. The characters include Ø, ® etc.
I have set my table to utf-8 as the default collation, all columns use table default, however when I try to insert these characters I get error: Incorrect string value: '\xEF\xBF\xBD' for column 'buyerName' at row 1
My connection string is defined as
string mySqlConn = "server="+server+";user="+username+";database="+database+";port="+port+";password="+password+";charset=utf8;";
I am at a loss as to why I am still seeing errors. Have I missed anything with either the .net connector, or with my MySQL setup?
--Edit--
My (new) C# insert statement looks like:
MySqlCommand insert = new MySqlCommand( "INSERT INTO fulfilled_Shipments_Data " +
"(amazonOrderId,merchantOrderId,shipmentId,shipmentItemId,"+
"amazonOrderItemId,merchantOrderItemId,purchaseDate,"+ ...
VALUES (#amazonOrderId,#merchantOrderId,#shipmentId,#shipmentItemId,"+
"#amazonOrderItemId,#merchantOrderItemId,#purchaseDate,"+
"paymentsDate,shipmentDate,reportingDate,buyerEmail,buyerName,"+ ...
insert.Parameters.AddWithValue("#amazonorderId",lines[0]);
insert.Parameters.AddWithValue("#merchantOrderId",lines[1]);
insert.Parameters.AddWithValue("#shipmentId",lines[2]);
insert.Parameters.AddWithValue("#shipmentItemId",lines[3]);
insert.Parameters.AddWithValue("#amazonOrderItemId",lines[4]);
insert.Parameters.AddWithValue("#merchantOrderItemId",lines[5]);
insert.Parameters.AddWithValue("#purchaseDate",lines[6]);
insert.Parameters.AddWithValue("#paymentsDate",lines[7]);
insert.ExecuteNonQuery();
Assuming that this is the correct way to use parametrized statements, it is still giving an error
"Incorrect string value: '\xEF\xBF\xBD' for column 'buyerName' at row 1"
Any other ideas?
\xEF\xBF\xBD is the UTF-8 encoding for the unicode character U+FFFD. This is a special character, also known as the "Replacement character". A quote from the wikipedia page about the special unicode characters:
The replacement character � (often a black diamond with a white question mark) is a symbol found in the Unicode standard at codepoint U+FFFD in the Specials table. It is used to indicate problems when a system is not able to decode a stream of data to a correct symbol. It is most commonly seen when a font does not contain a character, but is also seen when the data is invalid and does not match any character:
So it looks like your data source contains corrupted data. It is also possible that you try to read the data using the wrong encoding. Where do the lines come from?
If you can't fix the data, and your input indeed contains invalid characters, you could just remove the replacement characters:
lines[n] = lines[n].Replace("\xFFFD", "");
Mattmanser is right, never write a sql query by concatenating the parameters directly in the query. An example of parametrized query is:
string lastname = "Doe";
double height = 6.1;
DateTime date = new DateTime(1978,4,18);
var connection = new MySqlConnection(connStr);
try
{
connection.Open();
var command = new MySqlCommand(
"SELECT * FROM tblPerson WHERE LastName = #Name AND Height > #Height AND BirthDate < #BirthDate", connection);
command.Parameters.AddWithValue("#Name", lastname);
command.Parameters.AddWithValue("#Height", height);
command.Parameters.AddWithValue("#Name", birthDate);
MySqlDataReader reader = command.ExecuteReader();
...
}
finally
{
connection.Close();
}
To those who have a similar problem using PHP, try the function utf8_encode($string). It just works!
I have this some problem, when my website encoding is utf-u and I tried to send in form CP-1250 string (example taken by listdir dictionaries).
I think you must send string encoded like website.
I have a bunch of json escaped strings, for example
str = "what a war\/what a peace"
(the escape doesn't limit to slash "/")
I want to parse it to
"what a war/what a peace"
How can I do it with python 3.0 ?
Is there anything stopping your from using the built-in replace on string instances?
s = "what a war\/what a peace"
r = s.replace('\\', '')
print(r)
'what a war/what a peace'
As an addendum, I know this in an example of what you want to do but, refrain from using names like str, list et-cetera. You'll regret it later.
Duplicate headers received from server
The response from the server contained duplicate headers. This problem is generally the result of a misconfigured website or proxy. Only the website or proxy administrator can fix this issue.
Error 349 (net::ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION): Multiple distinct Content-Disposition headers received. This is disallowed to protect against HTTP response splitting attacks.
I found this error while exporting to pdf in chrome.
Response.Buffer = false;
Response.ClearHeaders();
string ext = objProp.PACKAGEFILENAME.Substring(objProp.PACKAGEFILENAME.LastIndexOf("."));
string ext1 = ext.Substring(1);
Response.ContentType = ext1;
Response.AddHeader("Content-Disposition", "target;_blank,attachment; filename=" + objProp.PACKAGEFILENAME);
const int ChunkSize = 1024;
byte[] binary = objProp.PACKAGEDOCUMENT;
System.IO.MemoryStream ms = new System.IO.MemoryStream(binary);
int SizeToWrite = ChunkSize;
for (int i = 0; i < binary.GetUpperBound(0) - 1; i = i + ChunkSize)
{
if (!Response.IsClientConnected) return;
if (i + ChunkSize >= binary.Length) SizeToWrite = binary.Length - i;
byte[] chunk = new byte[SizeToWrite];
ms.Read(chunk, 0, SizeToWrite);
Response.BinaryWrite(chunk);
Response.Flush();
}
Response.Close();
How to fix this?
This ones a little old but was high in the google ranking so I thought I would throw in the answer I found from Chrome, pdf display, Duplicate headers received from the server
Basically my problem also was that the filename contained commas. Do a replace on commas to remove them and you should be fine. My function to make a valid filename is below.
public static string MakeValidFileName(string name)
{
string invalidChars = Regex.Escape(new string(System.IO.Path.GetInvalidFileNameChars()));
string invalidReStr = string.Format(#"[{0}]+", invalidChars);
string replace = Regex.Replace(name, invalidReStr, "_").Replace(";", "").Replace(",", "");
return replace;
}
The server SHOULD put double quotes around the filename, as mentioned by #cusman and #Touko in their replies.
For example:
Response.AddHeader("Content-Disposition", "attachment;filename=\"" + filename + "\"");
Just put a pair of double quotes around your file name like this:
this.Response.AddHeader("Content-disposition", $"attachment; filename=\"{outputFileName}\"");
For me the issue was about a comma not in the filename but as below: -
Response.ok(streamingOutput,MediaType.APPLICATION_OCTET_STREAM_TYPE).header("content-disposition", "attachment, filename=your_file_name").build();
I accidentally put a comma after attachment. Got it resolved by replacing comma with a semicolon.
Double quotes around the filename in the header is the standard per MDN web docs. Omitting the quotes creates multiple opportunities for problems arising from characters in the filename.
I've got this same error (ERR_RESPONSE_HEADERS_MULTIPLE_CONTENT_DISPOSITION) downloading from EDGE a dynamic filename named
"a,b,c test.zip"
The problem was the comma.
Strangely, using the old "internet explorer", the comma was not a problem and the same file was downloaded (same procedure same file). Using other browser the error occured.
WORKAROUND: normalize filename (by code) replacing any comma into underscore, if evolved browsers are used.
In my application iOS I need to export some data into CSV or HTML format. How can I do this?
RegexKitLite comes with an example of how to read a csv file into an NSArray of NSArrays, and to go in the reverse direction is pretty trivial.
It'd be something like this (warning: code typed in browser):
NSArray * data = ...; //An NSArray of NSArrays of NSStrings
NSMutableString * csv = [NSMutableString string];
for (NSArray * line in data) {
NSMutableArray * formattedLine = [NSMutableArray array];
for (NSString * field in line) {
BOOL shouldQuote = NO;
NSRange r = [field rangeOfString:#","];
//fields that contain a , must be quoted
if (r.location != NSNotFound) {
shouldQuote = YES;
}
r = [field rangeOfString:#"\""];
//fields that contain a " must have them escaped to "" and be quoted
if (r.location != NSNotFound) {
field = [field stringByReplacingOccurrencesOfString:#"\"" withString:#"\"\""];
shouldQuote = YES;
}
if (shouldQuote == YES) {
[formattedLine addObject:[NSString stringWithFormat:#"\"%#\"", field]];
} else {
[formattedLine addObject:field];
}
}
NSString * combinedLine = [formattedLine componentsJoinedByString:#","];
[csv appendFormat:#"%#\n", combinedLine];
}
[csv writeToFile:#"/path/to/file.csv" atomically:NO];
The general solution is to use stringWithFormat: to format each row. Presumably, you're writing this to a file or socket, in which case you would write a data representation of each string (see dataUsingEncoding:) to the file handle as you create it.
If you're formatting a lot of rows, you may want to use initWithFormat: and explicit release messages, in order to avoid running out of memory by piling up too many string objects in the autorelease pool.
And always, always, always remember to escape the values correctly before passing them to the formatting method.
Escaping (along with unescaping) is a really good thing to write unit tests for. Write a function to CSV-format a single row, and have test cases that compare its result to correct output. If you have a CSV parser on hand, or you're going to need one, or you just want to be really sure your escaping is correct, write unit tests for the parsing and unescaping as well as the escaping and formatting.
If you can start with a single record containing any combination of CSV-special and/or SQL-special characters, format it, parse the formatted string, and end up with a record equal to the one you started with, you know your code is good.
(All of the above applies equally to CSV and to HTML. If possible, you might consider using XHTML, so that you can use XML validation tools and parsers, including NSXMLParser.)
CSV - comma separated values.
I usually just iterate over the data structures in my application and output one set of values per line, values within set separated with comma.
struct person
{
string first_name;
string second_name;
};
person tony = {"tony", "momo"};
person john = {"john", "smith"};
would look like
tony, momo
john, smith