I'm defining my custom model for pas.plugin.sqlalchemy.
Existing table of login details have binary field as password column. how should be define in model as binary. I tried to import _Binary class from SQLAlchemy after Googling but it doesn't import. Firstly I tried LargeBinary but when table been created its takes field type as Image which I dont want.
Any help would be appreciated.
Regards,
WEBBYFOX
I'm not surprised that _Binary doesn't work - even if it did, it's generally a very bad idea to use something that's intended as a private class. But "sqlalchemy.Binary" is a perfectly acceptable type.
That said, I doubt anybody's tested pas.plugins.sqlalchemy with Binary data for the password: PAS is a way to drop-in replacements for the existing Plone system, and out-of-the-box, Plone expects passwords to be strings.
Related
I have a Python script which collects data and sends it to my MySQL table.
I noticed that the "Cost" sometimes is 0,95 which results in 0 in my table since my table use "0.95" instead of "0,95".
I assume the best solution is to convert the , to . in my Python script by using:
variable.replace(",", ".")
However, couldn't one solution be to change format in my MySQL table? So that I store numbers in this format:
1100
0,95
0,1
150000
My Django Model
cost = models.DecimalField(max_digits=10, decimal_places=4, default=None)
Any feedback on how to best solve this issue?
Thanks
Your first instinct is correct: convert the "unusual" (comma-decimal) input into the standard format that MySQL used by default (dot-decimal) at the first point where you receive it.
there's lots of ways to write numbers
Be careful, though that you don't get stung by people using commas as thousands separators like "3,203,907.23", or the European form "3.203.907,23", the Swiss "3'203'907,23' or even this form, which is widely used in India: "32,03,907.71" (yes, I did mean to type only two digits there!)
To make your life easier, the rule for currencies is relatively simple:
where a dot or comma is followed by only two digits at the end of the string, that character is acting as the decimal separator.
Once you know which is the decimal separator, you can safely remove all other non-digits from the string, change the decimal separator you found to . then use any standard library string-to-number conversion.
Storage format isn't presentation format
Yes, you can tell MySQL to use comma as its decimal separator, but doing that will break so much of your code - including the parts of the framework that read from the database and expect dot-decimal numbers - that you'll regret doing it that way very quickly...
There's a general principle at work here: you should do your data storage and processing using a format that is easy to process, interchangeable with other systems, and understood by other software developers.
Consider what happens if you need to allow a different framework to access your MySQL database to generate reports... whoever develops that software (and it may be you) will be glad that the numbers are all stored the way numbers are "always" stored in databases.
Convert on the way in, re-convert on the way out
Where you need to accept input in a different format, convert that input into your standardised format as early as possible.
When you need to use an output format, do the conversion to that format as late as possible.
The idea is to keep as much of your system "unexceptional" as possible. A programmer who has to remember what numeric format will in force at the time when a given method is called is not a happy programmer.
P.S.
The option you're talking about in MySQL is an example of this pattern: it doesn't change how numeric data is stored. All that changes is how you pass numbers to MySQL and how it presents them back to you.
I am writing pdxInstances to GemFire using the sequence: rabbitmq => springxd => gemfire.
If I put this JSON into rabbitmq {'ID':11,'value':5}, value appears as a byte value in GemFire. If I put {'ID':11,'value':500}, value appears as a word and if I put {'ID':11,'value':50000} it appears as an Integer.
A problem arises when I query data from GemFire and order them. For example, if I use a query such as select * from /my_region order by value it fails, saying it cannot compare a byte with a word (or byte with an integer).
Is there any way to declare the data type in JSON? Or any other method to get rid of this problem?
To add a bit of insight into this problem... in reviewing GemFire/Geode source code, it would seem it is not possible to configure the desired value type and override GemFire/Geode's default behavior, which can be seen in JSONFormatter.setNumberField(..).
I will not explain how GemFire/Geode involves the JSONFormatter during a Region.put(key, value) operation as it is rather involved and beyond the scope of this discussion.
However, one could argue that the problem is not necessarily with the JSONFormatter class, since storing a numeric value in a byte is more efficient than storing the value in an integer, especially when the value would indeed fit into a byte. Therefore, the problem is really that the Comparator used in the Query processor should be able to compare numeric values in the same type family (byte, short, int, long), upcasting where appropriate.
If you feel so inclined, feel free to file a JIRA ticket in the Apache Geode JIRA repository at https://issues.apache.org/jira/browse/GEODE-72?jql=project%20%3D%20GEODE
Note, Apache Geode is the open source "core" of Pivotal GemFire now. See the Apache Geode website for more details.
Cheers!
Your best bet would be to take care of this with a custom module or a groovy script. You can either write a custom module in Java to do the conversion and then upload the custom module into SpringXD, then you could reference your custom module like any other processor. Or you could write a script in Groovy and pass the incoming data through a transform processor.
http://docs.spring.io/spring-xd/docs/current/reference/html/#processors
The actual conversion probably won't be too tricky, but will vary depending on which method you use. The stream creation would look something like this when you're done.
stream create --name myRabbitStream --definition "rabbit | my-custom-module | gemfire-json-server etc....."
stream create --name myRabbitStream --definition "rabbit | transform --script=file:/transform.groovy | gemfire-json-server etc...."
It seems like you have your source and sink modules set up just fine, so all you need to do is get your processor module setup to do the conversion and you should be all set.
I have a problem with JSON. Before we discovered error, column 'data' in mysql database had a type VARCHAR(255), where serialized json stored. About 2 months it worked well, but when project started to grow, 255 chars - became insufficient. But we forget to change type to TEXT. Now we have a problem, that our serialized json is stripped to 255 chars and now is invalid. I dont care about lost data, but I need to make minimal parsable/valid json.
for ex:
data = '{"state_id":[null,20],"dispatcher_id":[null,6057525],"uir":[null,{"level":"2"'
I need to make it valid, like this
data = '{"state_id":[null,20],"dispatcher_id":[null,6057525],"uir":[null,{"level":"2"}]}'
add }]} at the end of json.
Is there any quick way to do it? Or I should write my own parser/fixer?
You can use jsonlist
it's a pure implementation in JavaScript for validating JSON.
According to the validation you'll know what is missing in the end and add the missing value.
you can also check a demo online
I've done this task, so I want to share my solution
I have a weird encoding problem from my PyQt app to my mysql database.
I mean weird in the sense that it works in one case and not the other ones, even though I seem to be doing the exact same thing for all.
My process is the following:
I have some QFocusOutTextEdit elements in which I write text possibly containing accents and stuff (é,à,è,...)
I get the text written with :
text = self.ui.text_area.toPlainText()
text = text.toUtf8()
Then to insert it in my database I do :
text= str(text).decode('unicode_escape').encode('iso8859-1').decode('utf8')
I also set the character set of my database, the specific tables and the specific columns of the table to utf8.
It is working for one my text areas, and for the other ones it puts weird characters instead in my db.
Any hint appreciated on this !
RESOLVED :
sorry for the disturbance, apparently I had some fields in my database that weren't up to date and this was blocking the process of encoding somehow.
You are doing a lot of encoding, decoding, and reencoding which is hard to follow even if you know what all of it means. You should try to simplify this down to just working natively with Unicode strings. In Python 3 that means str (normal strings) and in Python 2 that means unicode (u"this kind of string").
Arrange for your connection to the MySQL database to use Unicode on input and output. If you use something high-level like Sqlalchemy, you probably don't need to do anything. If you use MySQLdb directly make sure you pass charset="UTF8" (which implies use_unicode) to the connect() method.
Then make sure the value you are getting from PyQT is a unicode value. I don't know PyQT. Check the type of self.ui.text_area or self.ui.text_area.toPlainText(). Hopefully it is already a Unicode string. If yes: you're all set. If no: it's a byte string which is probably encoded in UTF-8 so you can decode it with theresult.decode('utf8') which will give you a Unicode object.
Once your code is dealing with all Unicode objects and no more encoded byte strings, you don't need to do any kind of encoding or decoding anymore. Just pass the strings directly from PyQT to MySQL.
I am running RoR, Ruby 1.8.6/Rails 2.3.2.
I am persisting a String within a model object to MySQL to a Blob field. For a number of reasons, I have to use MySQL Blob data type instead of a Text data type, yet I want Rails to treat the field as a regular String (instead of a binary String - which is what I think it is doing?).
How can I force this? Everything works well in the display of the pages, the main issue is XML that is being produced.
A specific example is a "description" field of type Blob in MySQL displays perfectly in the HTML ERB, but when using the to_xml method on that model object the resulting XML is:
<description type="binary" encoding="base64">
VGhpcyB0d28tc2NyZWVuIGZvcm0gYWxsb3dzIGJ1c2luZXNzIGN1c3RvbWVy
cyB0byByYXRlIHRoZWlyIGxldmVsIG9mIHNhdGlzZmFjdGlvbiBhbmQgcmVx
dWVzdCB0byBzcGVhayB3aXRoIG1hbmFnZW1lbnQuIEl0IGlzIGVzcGVjaWFs
bHkgZGVzaWduZWQgZm9yIHRoZSBDb21taXNzaW9uIG9uIFZvY2F0aW9uYWwg
UmVoYWJpbGl0YXRpb24u
</description>
I just want it to appear be formatted as a normal string field like:
<description>test description</description>
What are the best ways to force this? I've been searching and reading through docs, but haven't been finding any helpful suggestions.
I appreciate any and all help,
thank you
I ended up monkey patching the XmlSerializer to treat binary data as text. I will never have to send binary data down in pure binary form via XML, so this was a valid option - not perfect, but in doing research couldn't find any way to override the default column settings.