In Analytics, if I try to Export > Share a Link to Query then a URL is copied to my clipboard.
It has the following structure:
https://analytics.applicationinsights.io/subscriptions/[subscription id]/resourcegroups/Default-ApplicationInsights-[region]/components/[resource]?q=[alphanumeric string]&apptype=web
The alphanumeric string is some sort of encoding of the actual query. Why do I say that? Because it grows or shrinks according to the size of the query. I tried seeing if it was Base64 or UUencode but neither worked. Also I tried 5 a's and 5 b's followed by 10 c's in the query (arbitrary query) to see if I would see a pattern but that didn't help either.
Some analysis with Unix tools showed that the alphanumeric string is a character set with 0-9, A-Z, +, /, and =.
Does anybody know this format so that I can make arbitrary query URLs?
Alternately being able to submit parameters to the query would solve my problem. My motivation is to link to Application Insights from my website and go to dynamic queries.
Examples of the encoded part:
Query: aaaaabbbbbbcccccccccccEncoding: ?q=H4sIAAAAAAAAA0tMBIIkMEhGAC4AHRlzExcAAAA%3D
Query: abcdefghijklmnopqrstuvwxyz0123456789
Encoding: ?q=H4sIAAAAAAAAA0tMSk5JTUvPyMzKzsnNyy8oLCouKS0rr6isMjA0MjYxNTO3sOQCANVo3%2FUlAAAA
There are 2 options to link a query.
Encoded query (works well for lengthy queries and special characters). The format is q=EncodedQuery. EncodedQuery is the query, encoded in the following way: (a) first it is compressed via gzip, and (b) then it is encoded using base64 encoding.
Plain text query. The format is query=QueryText. The downside is that the query length is (more) limited by browser's URL length limit. It may also not play well with special characters.
Hope that helps,
Yoram
Related
I am using MySql database and for a field I chose varchar(200) . To prevent issues I set on my html page maxlength 200 . So ideally there should be no problem . But If I let the user input 200 characters I get exception. So i tried and tried and only at 190 characters I can be sure it also fits the database. So in future I will to prevent issues always make size of varchar() 20% bigger than what user can input in html page.
May be carriage returns are considered 2 characters each when it comes to maxlength. Can you make sure, you have any carriage returns.
1\r\n
1\r\n
1\r\n
1
A varchar(200) should be able to store 200 characters. You shouldn't need to increase the size, but if you do, an arbitrary 10% increase won't guarantee to solve the problem unless you know what is causing it. The danger of an overflow will remain.
Some possible reasons that spring to mind:
As noted by #VigneshKumarA, it could be carriage returns being stored as two characters.
It could also be multibyte unicode characters -- ie anything other than the basic ASCII character set. If you're entering accented letters or symbols, or non-Latin scripts, they will take up more than one byte per character.
Escaped/encoded characters, if you are sanitising your data. For example if you're running htmlentities() or similar on the input string, you may be getting single characters from the input being converted into entity codes like &. This will obviously make the string longer than it was when input.
What I would recommend is that you use a database tool to examine the stored data and check to see why it is storing more characters than you expected. Understand what the discrepancy is caused by, and then either fix it or adapt your system to handle it so that you can be sure it will never overflow.
I'm working on a database import/export process in VB.NET which writes data from a MySQL (5.5) database to a plain text file. The application reads the data to a DataTable, then goes through the rows/columns to actually write the data to the OutputFile (System.IO.StreamWriter object). The encoding on the tables in this database is Latin1. There is a MediumBlob field in one of the tables I've been using for testing which contains image files stored as a byte array.
In my attempts to validate the output from my application, I've exported the data directly from the database using the MySQL Workbench, then compared that with the results I get when I write the same data from my application. In the direct export from MySQL Workbench, I see some of these bytes are exported with the backslash. When I read the data through my application, however, this escape character does not appear. Viewed through Notepad++, it clearly shows some distinct differences between the two output results (see screenshot).
Obviously, while apparently very similar, the two are not completely identical. My application is not including the backslashes for escaped characters, and some characters such as NULL are coming out differently altogether. My code for writing this field to the file is:
OutputFile.Write("'" & System.Text.Encoding.GetEncoding(28591).GetString(CType(COPYRow(ColumnIndex), Byte())) & "'")
There doesn't appear to be an overload for the GetString method that allows me to specify an escape character, so I'm wondering if there's another way that, using this method, I can ensure the characters are correctly encoded, including escape characters.
I'm "assuming" that this method should also work in general when I start working with my PostgreSQL database, but with possibly a different encoding. I'm trying to build things as "generic" as possible, but I'll have to worry about specifying encodings at run-time instead of hard-coding them later.
EDIT
I just ran across another SO question, which might point me in the right direction: Convert a Unicode string to an escaped ASCII string. Obviously, it might take a bit more work to get it right, but this looks like the closest thing to what I'm trying to accomplish.
I have to generate codes with custom fields: id of field+name of field+values of the field.
How long is the data I can encode inside the QRcode? I need to know how many fields\values I can insert.
Should I use XML or JSON or CSV? What is most generic and efficient?
XML / JSON will not qualify for a QR code's alphanumeric mode since it will include lower-case letters. You'll have to use byte mode. The max is 2,953 characters. But, the practical limit is far less -- perhaps a few hundred characters.
It is far better to encode a hyperlink to data if you can.
As Terence says, no reader will do anything with XML/JSON except show it. You need a custom reader anyway to do something useful with that data. (Which suggests this is not a good use case for QR codes.) But if you're making your own reader, you can use gzip compression to make the payload much smaller. Your reader would know to unzip it.
You might get away with something workable but this is not a good approach in general.
The maximum number of alphanumeric characters you can have is 4,296. Although this will require the lowest form of error correction and will be very hard to scan.
JSON is generally more efficient at data storage than XML.
However, you will need to write your own app to scan the code - I don't know of any which will process raw JSON or XML. All the scanners will show you the text, though.
I have a weird encoding problem from my PyQt app to my mysql database.
I mean weird in the sense that it works in one case and not the other ones, even though I seem to be doing the exact same thing for all.
My process is the following:
I have some QFocusOutTextEdit elements in which I write text possibly containing accents and stuff (é,à,è,...)
I get the text written with :
text = self.ui.text_area.toPlainText()
text = text.toUtf8()
Then to insert it in my database I do :
text= str(text).decode('unicode_escape').encode('iso8859-1').decode('utf8')
I also set the character set of my database, the specific tables and the specific columns of the table to utf8.
It is working for one my text areas, and for the other ones it puts weird characters instead in my db.
Any hint appreciated on this !
RESOLVED :
sorry for the disturbance, apparently I had some fields in my database that weren't up to date and this was blocking the process of encoding somehow.
You are doing a lot of encoding, decoding, and reencoding which is hard to follow even if you know what all of it means. You should try to simplify this down to just working natively with Unicode strings. In Python 3 that means str (normal strings) and in Python 2 that means unicode (u"this kind of string").
Arrange for your connection to the MySQL database to use Unicode on input and output. If you use something high-level like Sqlalchemy, you probably don't need to do anything. If you use MySQLdb directly make sure you pass charset="UTF8" (which implies use_unicode) to the connect() method.
Then make sure the value you are getting from PyQT is a unicode value. I don't know PyQT. Check the type of self.ui.text_area or self.ui.text_area.toPlainText(). Hopefully it is already a Unicode string. If yes: you're all set. If no: it's a byte string which is probably encoded in UTF-8 so you can decode it with theresult.decode('utf8') which will give you a Unicode object.
Once your code is dealing with all Unicode objects and no more encoded byte strings, you don't need to do any kind of encoding or decoding anymore. Just pass the strings directly from PyQT to MySQL.
I have a variable like say A= drug & medicare $12/$15.
I need to assign it to a text box, but only 'drug' is posted the server. The rest of the data gets truncated.
this.textbox.text= request.querystring["A"].tostring();
The following is not valid for a="foo&bar$12":
http://example.com?a=foo&bar$12
The & symbol is a reserved character, it seperates query string variables. You will need to percent encode a value before sending them to that page.
Also & is a reserved character in HTML/XML. I suggest reading up on percent encoding and html encoding.
I believe you have problems with HTML entities. You need to read up on HTML escaping in your tool of choice. & cannot stand in HTML, since it begins an entity sequence - it needs to be replaced with &. Without specifying at least which toolchain you're using (as per #Richard's comment), we can't really suggest the best way to do it.
EDIT: Now that I reread your question, it seems A is not a variable but a query parameter :) Reading comprehension fail. Anyway, in this case a similar problem exists: & is not a valid character for a query parameter, and it needs URL escaping. Again, how exactly to do it is in the documentation for your toolchain, but in essence & will need to be replaced by %26. Plus sign is also not permitted (or rather it has another meaning); others are tolerated (but there are nicer ways to write them).
That looks more or less like ASP.NET pseudocode, so I'm going to diagnose your problem as the query string needing to be URL encoded. Key/value pairs in the query string are separated by an ampersand (&), and ASP.NET (along with other web platforms) automatically parse out the key value pairs for you.
In this case, the ampersand terminates the value of the "A=..." key/value pair. The problem will be solved if you can URL encode the link that brings the user into your page. If actually using ASP.NET, you can use the HttpUtility.UrlEncode() method for that:
string myValue = Server.UrlEncode("drug & medicare $12/$15");
You'll end up with this querystring instead: A=drug%20%26%20medicare%20%2412%2F%2415