I have a bunch of json escaped strings, for example
str = "what a war\/what a peace"
(the escape doesn't limit to slash "/")
I want to parse it to
"what a war/what a peace"
How can I do it with python 3.0 ?
Is there anything stopping your from using the built-in replace on string instances?
s = "what a war\/what a peace"
r = s.replace('\\', '')
print(r)
'what a war/what a peace'
As an addendum, I know this in an example of what you want to do but, refrain from using names like str, list et-cetera. You'll regret it later.
Related
i have extracted the fix message as below from Unix server and now need to convert this message into JSON. how can we do this?
8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111
Note: I tried to make this a comment on the answer provided by #selbie, but the text was too long for a comment, so I am making it an answer.
#selbie's answer will work most of the time, but there are two edge cases in which it could fail.
First, in a tag=value field where the value is of type STRING, it is legal for value to contain the = character. To correctly cope with this possibility, the Java statement:
pair = item.split("=");
should be changed to:
pair = item.split("=", 2);
The second edge case is when there are a pair of fields, the first of which is of type LENGTH and the second is of type DATA. In this case, the value of the LENGTH fields specifies the length of the DATA field (without the delimiter), and it is legal for the value of the DATA field to contain the delimiter character (ASCII character 1, but denoted as | in both the question and Selbie's answer). Selbie's code cannot be modified in a trivial manner to deal with this edge case. Instead, you will need a more complex algorithm that consults a FIX data dictionary to determine the type of each field.
Since you didn't tag your question for any particular programming language, I'll give you a few sample solutions:
In javascript:
let s = "8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111"
let obj = {};
items = s.split("|")
items.forEach(item=>{
let pair = item.split("=");
obj[pair[0]] = pair[1];
});
let jsonString = JSON.stringify(obj);
Python:
import json
s = "8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111"
obj = {}
for item in s.split("|"):
pair = item.split("=")
obj[pair[0]] = pair[1]
jsonString = json.dumps(obj)
Porting the above solutions to other languages is an exercise for yourself. There's comments below about semantic ordering and handling cases where the the = or | chars are part of the content. That's on you to explore if you need to support those scenarios.
I get a json string drom my network reply it looks like
{\"topic\":\"ddd\",\"payload\":\"{\\\"eeee\\\":\\\"eee\\\",\\\"thingUID\\\":\\\"eee\\\",\\\"ee\\\":\\\"eee\\\",\\\"serialNumber\\\":\\\"eee\\\",\\\"epoch\\\":\\\"eeee\\\",\\\"eee\\\":\\\"eee\\\",\\\"eee\\\":\\\"eee\\\"}\",\"type\":\"eee\"}\n\n
how can i remove the best and fastest way all these backslahes "" in QT???
I do it like this
QByteArray data = _sseReply->readAll();
data = data.replace("\\\"", "\"");
data = data.replace("}\"", "}");
data = data.replace("\"{", "{");
any cooler solution available??
Just use QJsonDocument::fromJson(data).object() and don't need remove backslashes.
I've used the Facebook feature to download all my data. The resulting zip file contains meta information in JSON files. The problem is that unicode characters in strings in these JSON files are escaped in a weird way.
Here's an example of such a string:
"nejni\u00c5\u00be\u00c5\u00a1\u00c3\u00ad bod: 0 mnm Ben\u00c3\u00a1tky\n"
When I try parse the string for example with javascript's JSON.parse() and print it out I get:
"nejnižšà bod: 0 mnm Benátky\n"
While it should be
"nejnižší bod: 0 mnm Benátky\n"
I can see that \u00c5\u00be should somehow correspond to ž but I can't figure out the general pattern.
I've been able to figure out these characters so far:
'\u00c2\u00b0' : '°',
'\u00c3\u0081' : 'Á',
'\u00c3\u00a1' : 'á',
'\u00c3\u0089' : 'É',
'\u00c3\u00a9' : 'é',
'\u00c3\u00ad' : 'í',
'\u00c3\u00ba' : 'ú',
'\u00c3\u00bd' : 'ý',
'\u00c4\u008c' : 'Č',
'\u00c4\u008d' : 'č',
'\u00c4\u008f' : 'ď',
'\u00c4\u009b' : 'ě',
'\u00c5\u0098' : 'Ř',
'\u00c5\u0099' : 'ř',
'\u00c5\u00a0' : 'Š',
'\u00c5\u00a1' : 'š',
'\u00c5\u00af' : 'ů',
'\u00c5\u00be' : 'ž',
So what is this weird encoding? Is there any known tool that can correctly decode it?
The encoding is valid UTF-8. The problem is, JavaScript doesn't use UTF-8, it uses UTF-16. So you have to convert from the valid UTF-8, to JavaScript UTF-16:
function decode(s) {
let d = new TextDecoder;
let a = s.split('').map(r => r.charCodeAt());
return d.decode(new Uint8Array(a));
}
let s = "nejni\u00c5\u00be\u00c5\u00a1\u00c3\u00ad bod: 0 mnm Ben\u00c3\u00a1tky\n";
s = decode(s);
console.log(s);
https://developer.mozilla.org/docs/Web/API/TextDecoder
You can use a regular expression to find groups of almost unicode characters, decode them into Latin-1 and then encode back into UTF-8
The following code should work in python3.x:
import re
re.sub(r'[\xc2-\xf4][\x80-\xbf]+',lambda m: m.group(0).encode('latin1').decode('utf8'), s)
The JSON file itself is UTF-8, but the strings are UTF-16 characters converted to byte sequences then converted to UTF-8 using escape sequences.
This command fixes a file like this in Emacs:
(defun k/format-facebook-backup ()
"Normalize a Facebook backup JSON file."
(interactive)
(save-excursion
(goto-char (point-min))
(let ((inhibit-read-only t)
(size (point-max))
bounds str)
(while (search-forward "\"\\u" nil t)
(message "%.f%%" (* 100 (/ (point) size 1.0)))
(setq bounds (bounds-of-thing-at-point 'string))
(when bounds
(setq str (--> (json-parse-string (buffer-substring (car bounds)
(cdr bounds)))
(string-to-list it)
(apply #'unibyte-string it)
(decode-coding-string it 'utf-8)))
(setf (buffer-substring (car bounds) (cdr bounds))
(json-serialize str))))))
(save-buffer))
Thanks to Jen's excellent question and Shawn's comment.
Basically facebook seems to take each individual byte of the unicode string representation, then exporting to JSON as if these bytes are individual Unicode code points.
What we need to do is take last two characters of each sextet (e.g. c3 from \u00c3), concatenate them together and read as a Unicode string.
This is how I do it in Ruby (see gist):
require 'json'
require 'uri'
bytes_re = /((?:\\\\)+|[^\\])(?:\\u[0-9a-f]{4})+/
txt = File.read('export.json').gsub(bytes_re) do |bad_unicode|
$1 + eval(%Q{"#{bad_unicode[$1.size..-1].gsub('\u00', '\x')}"}).to_json[1...-1]
end
good_data = JSON.load(txt)
With bytes_re we catch all sequences of bad Unicode characters.
Then for each sequence replace '\u00' with '\x' (e.g. \xc3), put quotes around it " and use Ruby's built-in string parsing so that the \xc3\xbe... strings are converted to actual bytes, that will later remain as Unicode characters in the JSON or properly quoted by the #to_json method.
The [1...-1] is to remove quotes inserted by #to_json
I wanted to explain the code because question is not ruby specific and reader may use another language.
I guess somebody can do it with a sufficiently ugly sed command..
Just adding the general rule how to get from something like '\u00c5\u0098' to 'Ř'. Putting together the last two letters from the \u parts gets you c5 and 98 which are the two bytes of the utf-8 representation. UTF-8 encodes the code point in two bytes like this: 110xxxxx 10xxxxxx, where x are the actual bits of the character code. You can take the two bytes, use & to get the x parts, put them one after the next and read that as a number and you get the 0x158, which is the code for 'Ř'.
My javascript implementation:
function fixEncoding(s) {
var reg = /\\u00([a-f0-9]{2})\\u00([a-f0-9]{2})/gi;
return s.replace(reg, function(a, m1, m2){
b1 = parseInt(m1,16);
b2 = parseInt(m2,16);
var maskedb1 = b1 & 0x1F;
var maskedb2 = b2 & 0x3F;
var result = (maskedb1 << 6) | maskedb2;
return String.fromCharCode(result);
})
}
So... I've been working with WYSIWYG editors, and have realized, that they occasionally replace certain characters with the hex codes for that character, like the ' or the & for example.
How do I override String's index method such that it includes these hex codes?
Like, when do somestring.index("\'hello there") how do I get it to search \' and '
note: single quote is escaped for clarity against double quotes.
what is the most efficient way to do this kind of string search?
is there something like this already built in.
Also, since I'm using external tools, I don't really have a say in the format things are in.
THE SOLUTION:
search_reg_exp = Regexp.escape(str).gsub(/(both|options|or|more)/, "(both|options|or|more)")
long_str.index(search_reg_exp)
ORIGINAL ANSWER:
String#index doesn't just work for single characters, it can be used for a substring of any length, and you can give it a regular expression which would probably be best in this case:
some_string = "Russell's teapot"
another_string = "Russell's teapot"
apostrophe_expr = /'|'/
some_string.index apostrophe_expr
# => 7
another_string.index apostrophe_expr
# => 7
Another option would be to just decode the HTML entities before you start manipulating the string. There are various gems for this including html_helpers:
require 'html_helpers'
another_string = "Russell's teapot"
yet_another_string = HTML::EntityCoder.decode_entities another_string
# => "Russell's teapot"
yet_another_string.index "'"
# => 7
yet_another_string.index ?' # bonus syntax tip--Ruby 1.9.1+
# => 7
In my application iOS I need to export some data into CSV or HTML format. How can I do this?
RegexKitLite comes with an example of how to read a csv file into an NSArray of NSArrays, and to go in the reverse direction is pretty trivial.
It'd be something like this (warning: code typed in browser):
NSArray * data = ...; //An NSArray of NSArrays of NSStrings
NSMutableString * csv = [NSMutableString string];
for (NSArray * line in data) {
NSMutableArray * formattedLine = [NSMutableArray array];
for (NSString * field in line) {
BOOL shouldQuote = NO;
NSRange r = [field rangeOfString:#","];
//fields that contain a , must be quoted
if (r.location != NSNotFound) {
shouldQuote = YES;
}
r = [field rangeOfString:#"\""];
//fields that contain a " must have them escaped to "" and be quoted
if (r.location != NSNotFound) {
field = [field stringByReplacingOccurrencesOfString:#"\"" withString:#"\"\""];
shouldQuote = YES;
}
if (shouldQuote == YES) {
[formattedLine addObject:[NSString stringWithFormat:#"\"%#\"", field]];
} else {
[formattedLine addObject:field];
}
}
NSString * combinedLine = [formattedLine componentsJoinedByString:#","];
[csv appendFormat:#"%#\n", combinedLine];
}
[csv writeToFile:#"/path/to/file.csv" atomically:NO];
The general solution is to use stringWithFormat: to format each row. Presumably, you're writing this to a file or socket, in which case you would write a data representation of each string (see dataUsingEncoding:) to the file handle as you create it.
If you're formatting a lot of rows, you may want to use initWithFormat: and explicit release messages, in order to avoid running out of memory by piling up too many string objects in the autorelease pool.
And always, always, always remember to escape the values correctly before passing them to the formatting method.
Escaping (along with unescaping) is a really good thing to write unit tests for. Write a function to CSV-format a single row, and have test cases that compare its result to correct output. If you have a CSV parser on hand, or you're going to need one, or you just want to be really sure your escaping is correct, write unit tests for the parsing and unescaping as well as the escaping and formatting.
If you can start with a single record containing any combination of CSV-special and/or SQL-special characters, format it, parse the formatted string, and end up with a record equal to the one you started with, you know your code is good.
(All of the above applies equally to CSV and to HTML. If possible, you might consider using XHTML, so that you can use XML validation tools and parsers, including NSXMLParser.)
CSV - comma separated values.
I usually just iterate over the data structures in my application and output one set of values per line, values within set separated with comma.
struct person
{
string first_name;
string second_name;
};
person tony = {"tony", "momo"};
person john = {"john", "smith"};
would look like
tony, momo
john, smith