I have code below. Creating array of BYTEs, filling it with some sytes, and then trying to read it. When reading, i got an error:
int imageSize = 1024;
BYTE* input = new BYTE[imageSize];
// input is filling by some bytes..
DataWriter ^writer = ref new DataWriter();
writer->WriteBytes(ArrayReference<BYTE>(input, imageSize));
IBuffer ^buffer = writer->DetachBuffer();
DataReader ^ reader = DataReader::FromBuffer(buffer);
auto res = reader->ReadString(buffer->Length); // THIS STRING THROW Platform::COMException ^
Strings cannot contain random data - they must contain well-formed Unicode code-points. I assume the error (which you don't include) says this:
HRESULT:0x80070459 No mapping for the Unicode character exists in the target multi-byte code page.
From the name of your variable (imageSize) it sounds like you're trying to load bitmap data, which is not a valid String.
Related
I am trying to decrypt encrypted strs that are inside of a dictionary. To do this, the values must first be encoded, then the data can be decrypted and then decoded. I am struggling to encode the values without encoding the keys.
P.S: here is the printout for line 4:
b'{"pid": 6, "parent_first": "gAAAAABjAsAWbyxBsR804rVdMJ_Dzfj1s5s9GYVFSB2AJq3VSHTjH2V7lE4lt2gtO3LrhL6eKTm1qx153VO-g5xxWRb6mjXqvQ==", "parent_last": "gAAAAABjAsAWgiUupGo06d-tLY3WHgpfu5g0y55DjCPXdx4G2hIkEw50e3HAyi_r6z5NBHnJkevR8WkAy-2mhjvhUFRUk7Le8Q==", "email": "gAAAAABjAsAWAOTCver2_4bsBfrDA8SIdrykIH8Jojkd5100HT9y2Yz6ZnbZfYBqOYgwcKEquFhRGRZtey0A1Mdu12GxSHD3OdN8zb1DlLF0cP6O9tZEHGc=", "password": "f58a1612e9af7b5ee7e2141730f9a680f94765ad082918d489be42bde5d9ab23", "username": "9e7bd6851718b496e3c9cb0db480cbb8b87cf0455a7d627658906158357849b1"}'
for user in query:
dict = user.to_dict()
res_bytes = json.dumps(dict).encode('utf-8')
print(res_bytes)
res_bytes['parent_first'] = decrypt(res_bytes['parent_first'.encode('utf-8')])
res_bytes['parent_last'] = decrypt(res_bytes['parent_last'.encode('utf-8')])
res_bytes['email'] = decrypt(res_bytes['email'.encode('utf-8')])
res_bytes['username'] = decrypt(res_bytes['username'.encode('utf-8')])
dict = json.loads(res_bytes)
print(dict)
this gives an error of byte indices must be integers or slices, not bytes
res_bytes contains a byte sequence, so trying to work with it as if it's a dictionary instance doesn't make sense and results in the error you're seeing.
Basically, there's no reason to use json.dumps() and json.loads() in this code snippet.
Instead, try something like:
for user in query:
u = user.to_dict()
u['parent_first'] = decrypt(u['parent_first'])
u['parent_last'] = decrypt(u['parent_last'])
u['email'] = decrypt(u['email'])
u['username'] = decrypt(u['username'])
print(u)
A few notes:
Once you've converted user to a dictionary, proceed to use it as a dictionary.
Fernet.decrypt() can take either a str or bytes as input, so no need to worry about encoding/decoding it.
I am writing a web-application and am deserializing via jsony into norm-model-object types.
Norm-model-types are always ref objects. Somehow my code which is very similar to the default example in jsony's github documentation does not compile. Instead I receive the error SIGSEGV: Illegal storage access. (Attempt to read from nil?).
See here my code sample
import std/[typetraits, times]
import norm/[pragmas, model]
import jsony
const OUTPUT_TIME_FORMAT* = "yyyy-MM-dd'T'HH:mm:ss'.'ffffff'Z'"
type Character* {.tableName: "wikientries_character".} = ref object of Model
name*: string
creation_datetime*: DateTime
update_datetime*: DateTime
proc parseHook*(s: string, i: var int, v: var DateTime) =
##[ jsony-hook that is automatically called to convert a json-string to datetime
``s``: The full JSON string that needs to be serialized. Your type may only be a part of this
``i``: The index on the JSON string where the next section of it starts that needs to be serialized here
``v``: The variable to fill with a proper value]##
var str: string
s.parseHook(i, str)
v = parse(s, OUTPUT_TIME_FORMAT, utc())
proc newHook*(entry: var Character) =
let currentDateTime: DateTime = now()
entry.creation_datetime = currentDateTime # <-- This line is listed as the reason for the sigsev
entry.update_datetime = currentDateTime
entry.name = ""
var input = """ {"name":"Test"} """
let c = input.fromJson(Character)
I don't understand what the issue appears to be here, as the jsony-example on its github page looks pretty similar:
type
Foo5 = object
visible: string
id: string
proc newHook*(foo: var Foo5) =
# Populates the object before its fully deserialized.
foo.visible = "yes"
var s = """{"id":"123"}"""
var v = s.fromJson(Foo5)
doAssert v.id == "123"
doAssert v.visible == "yes"
How can I fix this?
The answer lies in the fact that norm-object-types are ref objects, not normal (value) objects (Thanks to ElegantBeef, Rika and Yardanico from the nim-discord to point this out)! If you do not explicitly 'create' a ref-type at one point, the memory for it is never allocated since the code doesn't do the memory allocation for you unlike with value types!
Therefore, you must initialize/create a ref-object first before you can use it, and Jsony does not take over initialization for you!
The correct way to write the above newHook thus looks like this:
proc newHook*(entry: var Character) =
entry = new(Character)
let currentDateTime: DateTime = now()
entry.creation_datetime = currentDateTime
entry.update_datetime = currentDateTime
entry.name = ""
Quick summary of issue - when I set the RawString to text, the email with attachment is sent and properly named. When I set the attachment to a RawString message, I get the error "The part 'PartAttachment' of message 'msg_Email' contained a null value at the end of the construct block."
I have created my own RawString class which I have used before. The orchestration receives a CSV file via a pass-thru pipline. Following some documentation I found, I receive a XmlDocument, and in the next shape, I construct the RawString as follows:
msg_Raw_String.MessagePart_1 = msg_Ledger6002_File_XmlDoc;
However if I set the to dummy text in RawString, the email with attachment is sent (the attachment is just a text file with the value below:
msg_Email.PartAttachment = new XXXLedger6002.Component.RawString("Test Text");
Full code in message assignment:
msg_Email.BodyPart = new XXXLedger6002.Component.RawString("See attached email. Method 2 Dynamic");
// Force some value to make sure it is not null
//msg_Email.PartAttachment = new Ledger6002.Component.RawString("Test Text");
// Tried both of these, same error:
//msg_Email.PartAttachment = msg_Ledger6002_File_XmlDoc;
msg_Email.PartAttachment = msg_Raw_String.MessagePart_1;
// syntax doesn't allow this:
// msg_Email.PartAttachment = msg_Raw_String;
// Set the filename as it should display on the attachment in the email
// (drop the path, just the filename/extension)
attachmentName = System.IO.Path.GetFileName(
msg_Ledger6002_File_XmlDoc(FILE.ReceivedFileName));
msg_Email.PartAttachment(MIME.FileName) = attachmentName;
//msg_Email.BodyPart(Microsoft.XLANGs.BaseTypes.ContentType) = "text/plain";//
//msg_Email.AttachmentPart(Microsoft.XLANGs.BaseTypes.ContentType) = "text/plain";
msg_Email(SMTP.Subject) = "Ledger6002 File";
msg_Email(SMTP.SMTPTo) = msg_Config_Email.smtpToEmail;
msg_Email(SMTP.From) = msg_Config_Email.smtpFromEmail;
msg_Email(SMTP.SMTPAuthenticate) = 0; // do not authenticate to SMTP server
msg_Email(SMTP.SMTPHost) = msg_Config_Email.smptHostName;
// msg_Email(SMTP.EmailBodyText) = "UTF-8";
msg_Email(SMTP.EmailBodyTextCharset) = "UTF-8";
msg_Email(SMTP.MessagePartsAttachments) = 2;
// No email generated with comment out line, trying same without mailto:
SMTP_Dyn_Email(Microsoft.XLANGs.BaseTypes.Address) = "mailto:" + msg_Config_Email.smtpToEmail;
SMTP_Dyn_Email(Microsoft.XLANGs.BaseTypes.TransportType) = "SMTP";
Error:
xlang/s engine event log entry: Uncaught exception (see the 'inner exception' below) has suspended an instance of service 'Ledger6002.Logic.Ledger6002_Process_File(9eb6993c-87d0-7bf0-b0bf-e1f684000af2)'.
The service instance will remain suspended until administratively resumed or terminated.
If resumed the instance will continue from its last persisted state and may re-throw the same unexpected exception.
InstanceId: ecaa7ed2-04c1-46b9-b845-cf3211b83387
Shape name: Send_Dyn_Email
ShapeId: 4ada59c3-367e-40b4-babc-d0d9999feb77
Exception thrown from: segment 1, progress 60
Inner exception: The part 'PartAttachment' of message 'msg_Email' contained a null value at the end of the construct block.
Exception type: NullPartException
Source: Microsoft.XLANGs.Engine
Target Site: System.IO.Stream Persist(System.String ByRef, Boolean)
The following is a stack trace that identifies the location where the exception occured
at Microsoft.XLANGs.Core.CustomFormattedPart.Persist(String& encoding, Boolean wantEncoding)
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.StagePartData(Part part)
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.PrepareMessage(XLANGMessage msg, IList promoteProps, IList toPromote)
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXXlangStore.WriteMessageState(IBTPEPInfoLookup pepLookup, Guid portId, XLANGMessage msg, Segment seg, String opname, String url, IList promoteProps, Boolean track, IList toPromote)
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXLogicalPortBinding.SendMessage(XLANGMessage msg, XlangStore store, Segment seg, OperationInfo op, IList additionalProps, IList toPromote, Boolean ignoreRoutingFailure)
at Microsoft.BizTalk.XLANGs.BTXEngine.BTXPortBase.SendMessage(Int32 iOperation, XLANGMessage msg, Correlation[] initCorrelations, Correlation[] followCorrelations, Context cxt, Segment seg, ActivityFlags flags)
at Ledger6002.Logic.Ledger6002_Process_File.segment1(StopConditions stopOn)
at Microsoft.XLANGs.Core.SegmentScheduler.RunASegment(Segment s, StopConditions stopCond, Exception& exp)
The following is a get-around that seems to work.
msg_Email.PartAttachment = new SBA.Ledger6002.Component.RawString(
msg_Raw_String.MessagePart_1.ToString());
If anybody can explain why I just can't set it to msg_Raw_String.MessagePart_1, I would love to know.
I am using libxml's HTML parser to create a dom tree of html documents. libxml gives text content of each node as a monolithic string (node), but my requirement is to further split each text node at spaces and create as many as word nodes. thus far I haven't found any options from libxml so I created a cpu expensive logic to split text nodes. Below is the part of recursive method that works.
void parse(xmlNodePtr cur, El*& parent) {
if (!cur) {
return;
}
string tagName = (const char*) cur->name;
string content = node_text(cur); // function defined below
Element* el = new Element(tagName, content);
parent->childs.push_back(el);
size_t pos;
string text;
cur = cur->children;
while (cur != NULL) {
if (xmlNodeIsText(cur) && (pos = node_text_find(cur, text, " ")) != string::npos) {
string first = text.substr(0, pos);
string second = text.substr(pos + 1);
El *el1 = new Element("text", first);
el->childs.push_back(el1);
El *el2 = new Element("text", " ");
el->childs.push_back(el2);
xmlNodeSetContent(cur, BAD_CAST second.c_str());
continue;
}
parse(cur, el);
cur = cur->next;
}
}
string node_text(xmlNodePtr cur) {
string content;
if (xmlNodeIsText(cur)) {
xmlChar *buf = xmlNodeGetContent(cur);
content = (const char*) buf;
}
return content;
}
size_t node_text_find(xmlNodePtr cur, string& text, string what){
text = node_text(cur);
return text.find_first_of(what);
}
The problem with above code is it didnt work for some UTF string like chinese language and moreover this code adds up time in overall parsing process.
Can anyone suggest a better way of doing this, thank you in advance !
I don't have a complete answer but I did see you doing explicit casts of xmlChar to char. That is a bad sign and probably why it doesn't work on Unicode.
If you're dealing with Unicode, which xmlChar probably is, you need to be using Unicode text processing libraries. Not std::string.
You actually have two choices. Find a library which processes in UTF-8 or convert UTF-8 into wchar (wide characters). If you convert to wchar then you can use wstring and its functions to process Unicode.
libxml2 xmlChar * to std::wstring looks like a useful answer.
As for speed, do my eyes deceive me or are you splitting on one space and creating a new element which you then split again? This is not the way to performance. I think it would go better if you remove the text node, split all of the words out and add the new nodes as you go.
The slowdown is most likely in the repeated creation, copying and destruction of objects. Work to minimize that. For example, if Element had a constructor form that accepted a begin/end iterator pair, or a start, length pair, that would be more efficient than creating a substring (copy!) and creating an Element (copy!) and then destroying the substrings.
The repeated calling of xmlNodeSetContent with the (probably large) second half of the text string, is giving you O2 performance. Not good.
In my application iOS I need to export some data into CSV or HTML format. How can I do this?
RegexKitLite comes with an example of how to read a csv file into an NSArray of NSArrays, and to go in the reverse direction is pretty trivial.
It'd be something like this (warning: code typed in browser):
NSArray * data = ...; //An NSArray of NSArrays of NSStrings
NSMutableString * csv = [NSMutableString string];
for (NSArray * line in data) {
NSMutableArray * formattedLine = [NSMutableArray array];
for (NSString * field in line) {
BOOL shouldQuote = NO;
NSRange r = [field rangeOfString:#","];
//fields that contain a , must be quoted
if (r.location != NSNotFound) {
shouldQuote = YES;
}
r = [field rangeOfString:#"\""];
//fields that contain a " must have them escaped to "" and be quoted
if (r.location != NSNotFound) {
field = [field stringByReplacingOccurrencesOfString:#"\"" withString:#"\"\""];
shouldQuote = YES;
}
if (shouldQuote == YES) {
[formattedLine addObject:[NSString stringWithFormat:#"\"%#\"", field]];
} else {
[formattedLine addObject:field];
}
}
NSString * combinedLine = [formattedLine componentsJoinedByString:#","];
[csv appendFormat:#"%#\n", combinedLine];
}
[csv writeToFile:#"/path/to/file.csv" atomically:NO];
The general solution is to use stringWithFormat: to format each row. Presumably, you're writing this to a file or socket, in which case you would write a data representation of each string (see dataUsingEncoding:) to the file handle as you create it.
If you're formatting a lot of rows, you may want to use initWithFormat: and explicit release messages, in order to avoid running out of memory by piling up too many string objects in the autorelease pool.
And always, always, always remember to escape the values correctly before passing them to the formatting method.
Escaping (along with unescaping) is a really good thing to write unit tests for. Write a function to CSV-format a single row, and have test cases that compare its result to correct output. If you have a CSV parser on hand, or you're going to need one, or you just want to be really sure your escaping is correct, write unit tests for the parsing and unescaping as well as the escaping and formatting.
If you can start with a single record containing any combination of CSV-special and/or SQL-special characters, format it, parse the formatted string, and end up with a record equal to the one you started with, you know your code is good.
(All of the above applies equally to CSV and to HTML. If possible, you might consider using XHTML, so that you can use XML validation tools and parsers, including NSXMLParser.)
CSV - comma separated values.
I usually just iterate over the data structures in my application and output one set of values per line, values within set separated with comma.
struct person
{
string first_name;
string second_name;
};
person tony = {"tony", "momo"};
person john = {"john", "smith"};
would look like
tony, momo
john, smith