I'm using Jinja2 and Google App Engine.
I identify my models with integer num:
class TestModel(db.Model):
value = db.StringProperty()
num = db.IntegerProperty()
and have a class for editing the value:
class Edit(Handler):
def get(self):
#show input box with value, let user edit value, render page with num
def post(self):
#get new value, filter database for model instance BY NUM and replace with new value
I need to store num on the page so I can retrieve it for the post method but I don't want the page to display it.
I tried:
<span name = "num" value = "{{num}}"></span>
and in the post method I have
num = self.request.get('num')
logging.error("NUM: %s" % num)
But when I run this, num doesn't show in the logs (I have "NUM: ") and I get
ValueError: invalid literal for int() with base 10: ''
When I render the page, the source code shows that num is there. Why isn't this code properly retrieving num?
Have you tried putting the field into a hidden input element. i.e.
<input type="hidden" name="..." value="..."/>
Also make sure that the value you are trying to retrieve in your post is within the form tags.
Related
I'm still a beginner with Scrapy, but this problem really got me scratching my head. I've got a webstore from which I need to extract data. The data is all on one page, but most of the time incomplete. It always has a name, but not always an amount or a description. It's structured in repeating classes like this. Note that this example has all three datafields filled.
I need:
The product name, located in < h4 class="mod-article-tile__title">
The product amount, located in < span class="price__unit">
The product description, located in < div class="mod-article-tile__info">
I managed to extract the data I need like this:
import pprint
import scrapy
class BasicSpider(scrapy.Spider):
name = 'aldi'
allowed_domains = ['aldi.nl']
base_url = 'https://www.aldi.nl/onze-producten/a-merken.html'
start_urls = ['https://www.aldi.nl/onze-producten/a-merken.html']
def parse(self, response):
products = response.xpath('//*[#class="mod-article-tile__content"]').extract()
name = response.xpath('//*[#class="mod-article-tile__title"]/text()').extract()
amount = response.xpath('//*[#class="price price--50 price--right mod-article-tile__price"]/text()').extract()
info = response.xpath('//*[#class="mod-article-tile__info"]/p/text()').extract()
i = 0
for product in products:
pprint.pprint(name[i] + " : " + amount[i] + ", " + info[i])
i+=1
However, this doesn't take incomplete data into account. So now since not all lists have the same length, an IndexError is thrown, and the data isn't assigned correctly. I tried parsing it using product, but I can't use xpath on it afterwards because it's a string.
So, is there a way to use xpath on the string result, or another way to extract the data from product? Or should I rather look into checking if the parsed data is empty, and insert empty data there?
Oh, and also, I can't seem to remove the pesky \n\t's that appear everywhere. I tried
def clean_string(self, string):
result = string.replace('\\n', '')
result = result.replace('\\t', '')
return result.strip()
But it didn't do the trick. Anyone able to drop a hint to resolve that?
the HTML CODE look like this:
<div class="WebPageMessage">
<div class="WebPageMessageInformation">
<div class="Message">Your request has been scheduled. Job # 221816. Click here To monitor Schedule<br>There are 0 job(s) already in the queue.</div></div></div>
Desired result: Extract the text "Job # 221816..."
here is the code im trying to use but i am getting "object is no supported" error msg
Set foo = objIE.document.getElementsByClassName("Message").getElementsByTagName("a").innerText
Any help would be great
Thanks and happy holidays : )
It's because you are attempting to Set a string (innerText) to an Object. Also, you need to specify which Class Name "Message" you want and which tag name "a" you want.
You have two options:
Option 1: Set the object first, then grab string from foo
Dim foo As Object, myStr As String
Set foo = objIE.document.getElementsByClassName("Message")(0).getElementsByTagName("a")(0)
myStr = foo.InnerText
Option 2: Don't assign the object to a variable and just grab the string directly.
Dim myStr As String
myStr = objIE.document.getElementsByClassName("Message")(0).getElementsByTagName("a")(0).innerText
Notice how I placed (0) after the ClassName and TagName? That is subject to change dependent entirely on your full HTML code. Best method would be using a For Each...Next statement to iterate through all the ClassNames or TagNames, but you can specify which Class/Tag Names you want specifically by placing (i) after. Notice the 's' after getElements...? That's because class and a tag names are not unique. The only unique element is the ID (notice no 's' in getElementByID().
In SqlAlchemy I use:
price = Column(Numeric(18, 5))
in various placed throught my app. When I get a number formatted in swedish, with a comma instead of a dot (0,34 instead of 0.34) and try to change the price column the number gets set to 0.00000.
To solve this I have this code:
obj.price = price.replace(',','.')
But having this all over the code makes it pretty ugly and the risk is that I forget one place. Would it be possible to have some kind of generic converter function which gets called before a value is converted from a string to a Numeric? And that I have that in one place only.
Check the validates decorator of SQLAlchemy: http://docs.sqlalchemy.org/en/rel_1_0/orm/mapped_attributes.html
A quick way to add a “validation” routine to an attribute is to use
the validates() decorator. An attribute validator can raise an
exception, halting the process of mutating the attribute’s value, or
can change the given value into something different.
In your case the code could look similar to:
from sqlalchemy.orm import validates
class Obj(Base):
__tablename__ = 'obj'
id = Column(Integer, primary_key=True)
price = Column(Numeric(18, 5))
#validates('price')
def validate_price(self, key, price):
if ',' in price:
return float(price.replace(',','.'))
else:
return float(price)
How do I update an HSTORE field with Flask-Admin?
The regular ModelView doesn't show the HSTORE field in Edit view. It shows nothing. No control at all. In list view, it shows a column with data in JSON notation. That's fine with me.
Using a custom ModelView, I can change the HSTORE field into a TextAreaField. This will show me the HSTORE field in JSON notation when in edit view. But I cannot edit/update it. In list view, it still shows me the object in JSON notation. Looks fine to me.
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
When I attempt to save/edit the JSON, I receive this error:
sqlalchemy.exc.InternalError
InternalError: (InternalError) Unexpected end of string
LINE 1: UPDATE mytable SET attributes='{}' WHERE mytable.id = ...
^
'UPDATE mytable SET attributes=%(attributes)s WHERE mytable.id = %(mytable_id)s' {'attributes': u'{}', 'mytable_id': 14L}
Now -- using code, I can get something to save into the HSTORE field:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
model.attributes = {"a": "1"}
return
This basically overrides the model and put this object into it. I can then see the object in the List view and the Edit view. Still not good enough -- I want to save/edit the object that the user typed in.
I tried to parse and save the content from the form into JSON and back out. This doesn't work:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
x = form.data['attributes']
y = json.loads(x)
model.attributes = y
return
json.loads(x) says this:
ValueError ValueError: Expecting property name: line 1 column 1 (char
1)
and here are some sample inputs that fail:
{u's': u'ff'}
{'s':'ff'}
However, this input works:
{}
Blank also works
This is my SQL Table:
CREATE TABLE mytable (
id BIGSERIAL UNIQUE PRIMARY KEY,
attributes hstore
);
This is my SQA Model:
class MyTable(Base):
__tablename__ = u'mytable'
id = Column(BigInteger, primary_key=True)
attributes = Column(HSTORE)
Here is how I added the view's to the admin object
admin.add_view(ModelView(models.MyTable, db.session))
Add the view using a custom Model View
admin.add_view(MyView(models.MyTable, db.session))
But I don't do those views at the same time -- I get a Blueprint name collision error -- separate issue)
I also attempted to use a form field converter. I couldn't get it to actually hit the code.
class MyModelConverter(AdminModelConverter):
def post_process(self, form_class, info):
raise Exception('here I am') #but it never hits this
return form_class
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
The answer gives you a bit more then asked
Fist of all it "extends" hstore to be able to store actually JSON, not just key-value
So this structure is also OK:
{"key":{"inner_object_key":{"Another_key":"Done!","list":["no","problem"]}}}
So, first of all your ModelView should use custom converter
class ExtendedModelView(ModelView):
model_form_converter=CustomAdminConverter
Converter itself should know how to use hstore dialect:
class CustomAdminConverter(AdminModelConverter):
#converts('sqlalchemy.dialects.postgresql.hstore.HSTORE')
def conv_HSTORE(self, field_args, **extra):
return DictToHstoreField(**field_args)
This one as you can see uses custom WTForms field which converts data in both directions:
class DictToHstoreField(TextAreaField):
def process_data(self, value):
if value is None:
value = {}
else:
for key,obj in value.iteritems():
if (obj.startswith("{") and obj.endswith("}")) or (obj.startswith("[") and obj.endswith("]")):
try:
value[key]=json.loads(obj)
except:
pass #
self.data=json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
for key,obj in self.data.iteritems():
if isinstance(obj,dict) or isinstance(obj,list):
self.data[key]=json.dumps(obj)
if isinstance(obj,int):
self.data[key]=str(obj)
The final step will be to actual use this data in application
I did not make it in common nice way for SQLalchemy, since was used with flask-restful, so I have only adoption for flask-restful in one direction, but I think it's easy to get the idea from here and do the rest.
And if your case is simple key-value storage so nothing additionaly should be done, just use it as is.
But if you want to unwrap JSON somewhere in code, it's simple like this whenever you use it, just wrap in function
if (value.startswith("{") and value.endswith("}")) or (value.startswith("[") and value.endswith("]")):
value=json.loads(value)
Creating dynamical field for actual nice non-JSON way for editing of data also possible by extending FormField and adding some javascript for adding/removing fields, but this is whole different story, in my case I needed actual json storage, with blackjack and lists :)
Was working on postgres JSON datatype. The above solution worked great with a minor modifications.
Tried
'sqlalchemy.dialects.postgresql.json.JSON',
'sqlalchemy.dialects.postgresql.JSON',
'dialects.postgresql.json.JSON',
'dialects.postgresql.JSON'
The above versions did not work.
Finally the following change worked
#converts('JSON')
And changed class DictToHstoreField to the following:
class DictToJSONField(fields.TextAreaField):
def process_data(self, value):
if value is None:
value = {}
self.data = json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
else:
self.data = '{}'
Although, this is might not be the answer to your question, but by default SQLAlchemy's ORM doesn't detect in-place changes to HSTORE field values. But fortunately there's a solution: SQLAlchemy's MutableDict type:
from sqlalchemy.ext.mutable import MutableDict
class MyClass(Base):
__tablename__ = 'mytable'
id = Column(Integer, primary_key=True)
attributes = Column(MutableDict.as_mutable(HSTORE))
Now when you change something in-place:
my_object.attributes.['some_key'] = 'some value'
The hstore field will be updated after session.commit().
I had a similar question before, but this is a few steps beyond that so here we go:
I am trying to access values in a category in Tridion 2011. I am using the Razor TBB and using this code to do it:
#foreach (var keyword in Publication.MetaData.myCategory) {
#: Hello World!
}
I have set up a metadata schema with a field that has an xml name of "myCategory" attached to my publication. If I run this on the publication where myCategory is a Text field, this code works... kind of. It treats EACH character as a separate value of the keyword variable... so if I enter the text "one", what prints is "Hello World! Hello World! Hello World!", and if I just have "o" as the value, it prints "Hello World!".
Bizarre as that is (and I'd like to know why on that too), what I really want the field to be a "Values will be selected from a list" type of field, pointing to my category in Categories and keywords. When I do this, and the value of myCategory changes to the value of the item selected in the dropdown for this type of field instead of direct text entry, the code no longer works and gives this error:
Cannot implicitly convert type 'Tridion.Extensions.Mediators.Razor.Models.KeywordModel' to 'System.Collections.IEnumerable'. An explicit conversion exists (are you missing a cast?)
Please help me.
If you get your output repeated for every character in a string, you are clearly iterating over the characters in the string and not the other range of values that you expected.
I quickly checked the relevant code of the Razor mediator and its ToString method indeed returns the Title of the underlying RepositoryLocalObject.
http://code.google.com/p/razor-mediator-4-tridion/source/browse/trunk/Tridion.Extensions.Mediators.RazorMediator/Tridion.Extensions.Mediators.RazorMediator/Models/AbstractRepositoryLocalObject.cs
http://code.google.com/p/razor-mediator-4-tridion/source/browse/trunk/Tridion.Extensions.Mediators.RazorMediator/Tridion.Extensions.Mediators.RazorMediator/Models/KeywordModel.cs
http://code.google.com/p/razor-mediator-4-tridion/source/browse/trunk/Tridion.Extensions.Mediators.RazorMediator/Tridion.Extensions.Mediators.RazorMediator/Models/DynamicItemFields.cs
The code that handles KeywordFields in in the DynamicItemsFields.cs file:
else if (itemField is KeywordField)
{
KeywordField keywordField = (KeywordField)itemField;
if (keywordField.Definition.MaxOccurs == 1)
if (keywordField.Value == null)
_dictionary[key] = null;
else
_dictionary[key] = new KeywordModel(_engine, keywordField.Value);
else
{
List<KeywordModel> keywords = new List<KeywordModel>();
int i = 0;
foreach (Keyword k in keywordField.Values)
{
var kw = new KeywordModel(_engine, k);
kw.Index = i++;
kw.IsLast = Index == keywordField.Values.Count - 1;
keywords.Add(kw);
}
_dictionary[key] = keywords;
}
}
So it looks like the myCategory property will either be a KeywordModel object (if the KeywordField is single-value) or a List<KeywordModel> (if the KeywordField is marked as multi-value in the Schema). Is your myCategory field single value? Or multi-value?
If it is single-value, what type of output were your expecting? If you were expecting the list of allowed values (instead of the currently selected value), check if you can access it through myCategory.Definition somehow (which should be a regular TOM.NET KeywordFieldDefinition object).