How to have an argument possible values dependent on another argument with IPython Widgets? - widget

Assume I have this simple function in Python:
def f(gender, name):
if gender == 'male':
return ranking_male(name)
else:
return ranking_female(name)
where gender belongs to ['male', 'female'] whereas name belongs to ['Adam', 'John', 'Max', 'Frodo'] (if gender is male) or ['Mary', 'Sarah', 'Arwen'] (otherwise).
I wish to apply interact from ipywidgets to this function f. Normally one would do
from ipywidgets import interact
interact(f, gender = ('male', 'female'), name = ('Adam', 'John', 'Max', 'Frodo'))
The problem is that the admissible values for name now depend on the value chosen for gender.
I tried to find it in the docs but couldn't find it. The only thing I think may be important is
This is used to setup dynamic notifications of trait changes.
Parameters
----------
handler : callable
A callable that is called when a trait changes. Its
signature should be ``handler(change)``, where ``change```is a
dictionary. The change dictionary at least holds a 'type' key.
* ``type``: the type of notification.
Other keys may be passed depending on the value of 'type'. In the
case where type is 'change', we also have the following keys:
* ``owner`` : the HasTraits instance
* ``old`` : the old value of the modified trait attribute
* ``new`` : the new value of the modified trait attribute
* ``name`` : the name of the modified trait attribute.
names : list, str, All
If names is All, the handler will apply to all traits. If a list
of str, handler will apply to all names in the list. If a
str, the handler will apply just to that name.
type : str, All (default: 'change')
The type of notification to filter by. If equal to All, then all
notifications are passed to the observe handler.
But I have no idea how to do it nor to interpret what the doc string is talking about. Any help is much appreciated!

For example you have brand and model of car and model depends on brand.
d = {'Volkswagen' : ['Tiguan', 'Passat', 'Polo', 'Touareg', 'Jetta'], 'Chevrolet' : ['TAHOE', 'CAMARO'] }
brand_widget = Dropdown( options=list(d.keys()),
value='Volkswagen',
description='Brand:',
style=style
)
model_widget = Dropdown( options=d['Volkswagen'],
value=None,
description='Model:',
style=style
)
def on_update_brand_widget(*args):
model_widget.options = d[brand_widget.value]
brand_widget.observe(on_update_brand_widget, 'value')

I've used nested widgets to solve this problem. It'll work, but it's ugly, partially because it doesn't seem to be a common use case in ipywidgets (see discussion).
Given your function f(gender, name) you can define an intermediate wrapper:
def f_intermediate_wrapper(gender):
if gender=="male":
possible_names = ['Adam', 'John', 'Max', 'Frodo']
else:
possible_names = ['Mary', 'Sarah', 'Arwen']
try:
f_intermediate_wrapper.name_widget.widget.close()
except AttributeError:
pass
f_intermediate_wrapper.name_widget = interact(f,
gender=widgets.fixed(gender),
name = possible_names)
The first part sets the possible name options given the gender, as desired.
The second part closes the name_widget from your previous evaluation, if it exists. Otherwise, every time you change the gender, it'll leave up the old list of names, which are the wrong gender (see example).
The third part creates a name widget of the possible names for that gender, and stores it somewhere sufficiently static. (Otherwise, when you change the gender the old name widget will be out of scope, and you won't be able to close it.)
Now you can create your gender and name widget:
gender_and_name_widget = interact(f_intermediate_wrapper,
gender = ["male", "female"])
And you can access the result of your f(gender, name) using
gender_and_name_widget.name_widget.widget.result

Related

How to access json schema info in SparkDataset Api when using plain map/reduce functions?

Given there is a dataset of messages, defined by following code:
case class Message(id: Int, value: String)
var messages = Seq(
(0, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}"""),
(1, """{"action":"update","timestamp":"2017-10-05T23:01:19Z"}""")
).toDF("id", "value").as[Message]
var schema = new StructType().add("action", StringType).add("timestamp", TimestampType)
var res = messages.select(
from_json(col("value").cast("string"), schema)
)
+------------------------------------+
|jsontostructs(CAST(value AS STRING))|
+------------------------------------+
| [update,2017-10-0...|
| [update,2017-10-0...|
What is the best way to access the schema information in a plain map function. The function itself returns a row which has lost all the Type infos. In order to reach to the values one has to specify the type again e.g
res.head().getStruct(0).getValuesMap[TimestampType](Seq("timestamp"))
=> Map[String,org.apache.spark.sql.types.TimestampType] = Map(timestamp -> 2017-10-06 01:01:19.0)
or
res.head().getStruct(0).getString(0)
=> res20: String = update
Is there some better way to access the raw json data without spark sql aggregation functions?
As a rule of thumb:
To use collection API (map, flatMap, mapPartitions, groupByKey, etc.) use strongly typed API - define record type (case class works the best) which reflects the schema and use Encoders to convert things back and forth:
case class Value(action: String, timestamp: java.sql.Timestamp)
case class ParsedMessage(id: Int, value: Option[Value])
messages.select(
$"id", from_json(col("value").cast("string"), schema).alias("value")
).as[ParsedMessage].map(???)
With Dataset[Row] stay with high level SQL / DataFrame API (select, where, agg, groupBy)

In Ada, How do I recursively map and memory manage a type within itself

I've been struggling with this little issue for a while. I am trying to create my own implementation of an internal JSON structure. The challenge is that with Ada I have to use an access type to make it recursive and access types have the risk of leaking if I don't have it tightly controlled. In order to make it controlled, I kept all the real activity private I provided Get (Source:...) and Set (Target:...; Value:...) functions/procedures for the Node type that will attempt to verify and handle any existing Vector (json-array) or Map (json-object) elements. In order to further ensure that I was using stable features of Ada 2012 and catching contents as they go out of scope, I tried to use a Protected_Controlled type and "managing" Ada libraries, but found that the container libraries couldn't handle protected types, so I used simply Controlled. The Finalize (...) procedure is for any Vector or Map types and recursively frees the Node_Value.Reference.
My question is if I am applying Ada 2012 correctly, or else how do I create a memory managed recursion of a type that could be either a vector/map or a string/number?
private
...
type Node_Access is access Node;
type Node_Value is new Ada.Finalization.Controlled with record
Reference : Node_Access;
end record;
overriding procedure Initialize (Item : in out Node_Value);
overriding procedure Adjust (Item : in out Node_Value);
overriding procedure Finalize (Item : in out Node_Value);
...
package Of_Array is new Ada.Containers.Indefinite_Vectors (Natural, Node_Value);
package Of_Object is new Ada.Containers.Indefinite_Ordered_Maps (Wide_String, Node_Value);
type Node is record
...
Vector : aliased Of_Array.Vector;
Object : aliased Of_Object.Map;
end record
with Size => 96;
procedure Free is new Ada.Unchecked_Deallocation (Node, Node_Access);
The way to do it (in my opinion) is to use OOP and have an abstract element as the root node of a family of types representing the different kinds of data which can be stored.
An array of elements can then be implemented as a vector of the class rooted at the abstract element type. An "object" can be implemented as a hash-table with a string key and the class rooted at the abstract element type as the values.
Self-referential types without access types are a valid use for type extension in combination with an indefinite container. A simple example is S-expressions, or Sexes. A Sex is either an atom or a list of zero or more Sexes. The right way to be able to do this would be
with Ada.Containers.Indefinite_Vectors;
package Sexes is
type Sex is private;
-- Operations on Sex
private -- Sexes
package Sex_List is new Ada.Containers.Indefinite_Vectors
(Index_Type => Positive, Element_Type => Sex); -- Illegal
type Sex (Is_Atom : Boolean := False) is record
case Is_Atom is
when False =>
Value : Atom;
when True =>
List : Sex_List.Vector;
end case;
end record;
end Sexes;
but Ada doesn't allow this. We can use type extension to get around this:
private -- Sexes
type Root is tagged null record;
package Sex_List is new Ada.Containers.Indefinite_Vectors
(Index_Type => Positive, Element_Type => Root'Class);
type Sex (Is_Atom : Boolean := False) is new Root with record
case Is_Atom is
when False =>
Value : Atom;
when True =>
List : Sex_List.Vector;
end case;
end record;
end Sexes;
which is legal. The only catch is that you have to convert anything taken from List to Sex (or Node in your case).
HTH; sorry about the late response.

Modify all numbers before insert or update

In SqlAlchemy I use:
price = Column(Numeric(18, 5))
in various placed throught my app. When I get a number formatted in swedish, with a comma instead of a dot (0,34 instead of 0.34) and try to change the price column the number gets set to 0.00000.
To solve this I have this code:
obj.price = price.replace(',','.')
But having this all over the code makes it pretty ugly and the risk is that I forget one place. Would it be possible to have some kind of generic converter function which gets called before a value is converted from a string to a Numeric? And that I have that in one place only.
Check the validates decorator of SQLAlchemy: http://docs.sqlalchemy.org/en/rel_1_0/orm/mapped_attributes.html
A quick way to add a “validation” routine to an attribute is to use
the validates() decorator. An attribute validator can raise an
exception, halting the process of mutating the attribute’s value, or
can change the given value into something different.
In your case the code could look similar to:
from sqlalchemy.orm import validates
class Obj(Base):
__tablename__ = 'obj'
id = Column(Integer, primary_key=True)
price = Column(Numeric(18, 5))
#validates('price')
def validate_price(self, key, price):
if ',' in price:
return float(price.replace(',','.'))
else:
return float(price)

Define a relationship() that is only true sometimes

I'm working with a database schema that has a relationship that isn't always true, and I'm not sure how to describe it with sqlalchemy's ORM.
All the primary keys in this database are stored as a blob type, and are 16 byte binary strings.
I have a table called attribute, and this table has a column called data_type. There are a number of built in data_types, that are not defined explicitly in the database. So, maybe a data_type of 00 means it is a string, and 01 means it is a float, etc (those are hex values). The highest value for the built in data types is 12 (18 in decimal).
However, for some rows in attribute, the value of the attribute stored in the row must exist in a pre-defined list of values. In this case, data_type referrs to lookup.lookup_id. The actual data type for the attribute can then be retrieved from lookup.data_type.
I'd like to be able to call just Attribue.data_type and get back 'string' or 'number'. Obviously I'd need to define the {0x00: 'string', 0x01: 'number'} mapping somewhere, but how can I tell sqlalchemy that I want lookup.data_type if the value of attribute.data_type is greater than 18?
There are a couple of ways to do this.
The simplest, by far, is to just put your predefined data types into the table lookup. You say that you "need to define the... mapping somewhere", and a table is as good a place as any.
Assuming that you can't do that, the next simplest thing is to create a python property on class Attribute. The only problem will be that you can't query against it. You'll want to reassign the column data_type so that it maps to _data_type:
data_type_dict = {0x00: 'string',
0x01: 'number,
...}
class Attribute(Base):
__tablename__ = 'attribute'
_data_type = Column('data_type')
...
#property
def data_type(self):
dt = data_type_dict.get(self._data_type, None)
if dt is None:
s = Session.object_session(self)
lookup = s.query(Lookup).filter_by(id=self._data_type).one()
dt = lookup.data_type
return dt
If you want this to be queryable, that is, if you want to be able to do session.query(Attribute).filter_by(data_type='string'), you need to map data_type to something the database can handle, i.e., an SQL statement. You could do this in raw SQL as a CASE expression:
from sqlalchemy.sql.expression import select, case
class Attribute(Base):
...
data_type = column_property(select([attribute, lookup])\
.where(attribute.data_type==lookup.lookup_id)\
.where(case([(attribute.data_type==0x00, 'string'),
(attribute.data_type==0x01, 'number'),
...],
else_=lookup.data_type))
I'm not 100% certain that last part will work; you may need to explicitly join the tables attribute and lookup to specify that it's an outer join, though I think SQLAlchemy does that by default. The downside of this approach is that you are always going to try to join with the table lookup, though to query using SQL, you sort of have to do that.
The final option is to use a polymorphism, and map the two cases (data_type greater/less than 18) to two different subclasses:
class Attribute(Base):
__tablename__ = 'attribute'
_data_type = Column('data_type')
_lookup = column_property(attribute.data_type > 18)
__mapper_args__ = {'polymorphic_on': _lookup}
class FixedAttribute(Attribute):
__mapper_args__ = {'polymorphic_identity': 0}
data_type = column_property(select([attribute.data_type])\
.where(case([(attribute.data_type==0x00, 'string'),
(attribute.data_type==0x01, 'number'),
...])))
class LookupAttribute(Attribute):
__mapper_args__ = {'polymorphic_identity': 1}
data_type = column_property(select([lookup.data_type],
whereclause=attribute.data_type==lookup.lookup_id))
You might have to replace the 'polymorphic_on': _lookup with an explicit attribute.data_type > 18, depending on when that ColumnProperty gets bound.
As you can see, these are all really messy. Do #1 if it's at all possible.

How do we update an HSTORE field with Flask-Admin?

How do I update an HSTORE field with Flask-Admin?
The regular ModelView doesn't show the HSTORE field in Edit view. It shows nothing. No control at all. In list view, it shows a column with data in JSON notation. That's fine with me.
Using a custom ModelView, I can change the HSTORE field into a TextAreaField. This will show me the HSTORE field in JSON notation when in edit view. But I cannot edit/update it. In list view, it still shows me the object in JSON notation. Looks fine to me.
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
When I attempt to save/edit the JSON, I receive this error:
sqlalchemy.exc.InternalError
InternalError: (InternalError) Unexpected end of string
LINE 1: UPDATE mytable SET attributes='{}' WHERE mytable.id = ...
^
'UPDATE mytable SET attributes=%(attributes)s WHERE mytable.id = %(mytable_id)s' {'attributes': u'{}', 'mytable_id': 14L}
Now -- using code, I can get something to save into the HSTORE field:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
model.attributes = {"a": "1"}
return
This basically overrides the model and put this object into it. I can then see the object in the List view and the Edit view. Still not good enough -- I want to save/edit the object that the user typed in.
I tried to parse and save the content from the form into JSON and back out. This doesn't work:
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
def on_model_change(self, form, model, is_created):
x = form.data['attributes']
y = json.loads(x)
model.attributes = y
return
json.loads(x) says this:
ValueError ValueError: Expecting property name: line 1 column 1 (char
1)
and here are some sample inputs that fail:
{u's': u'ff'}
{'s':'ff'}
However, this input works:
{}
Blank also works
This is my SQL Table:
CREATE TABLE mytable (
id BIGSERIAL UNIQUE PRIMARY KEY,
attributes hstore
);
This is my SQA Model:
class MyTable(Base):
__tablename__ = u'mytable'
id = Column(BigInteger, primary_key=True)
attributes = Column(HSTORE)
Here is how I added the view's to the admin object
admin.add_view(ModelView(models.MyTable, db.session))
Add the view using a custom Model View
admin.add_view(MyView(models.MyTable, db.session))
But I don't do those views at the same time -- I get a Blueprint name collision error -- separate issue)
I also attempted to use a form field converter. I couldn't get it to actually hit the code.
class MyModelConverter(AdminModelConverter):
def post_process(self, form_class, info):
raise Exception('here I am') #but it never hits this
return form_class
class MyView(ModelView):
form_overrides = dict(attributes=fields.TextAreaField)
The answer gives you a bit more then asked
Fist of all it "extends" hstore to be able to store actually JSON, not just key-value
So this structure is also OK:
{"key":{"inner_object_key":{"Another_key":"Done!","list":["no","problem"]}}}
So, first of all your ModelView should use custom converter
class ExtendedModelView(ModelView):
model_form_converter=CustomAdminConverter
Converter itself should know how to use hstore dialect:
class CustomAdminConverter(AdminModelConverter):
#converts('sqlalchemy.dialects.postgresql.hstore.HSTORE')
def conv_HSTORE(self, field_args, **extra):
return DictToHstoreField(**field_args)
This one as you can see uses custom WTForms field which converts data in both directions:
class DictToHstoreField(TextAreaField):
def process_data(self, value):
if value is None:
value = {}
else:
for key,obj in value.iteritems():
if (obj.startswith("{") and obj.endswith("}")) or (obj.startswith("[") and obj.endswith("]")):
try:
value[key]=json.loads(obj)
except:
pass #
self.data=json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
for key,obj in self.data.iteritems():
if isinstance(obj,dict) or isinstance(obj,list):
self.data[key]=json.dumps(obj)
if isinstance(obj,int):
self.data[key]=str(obj)
The final step will be to actual use this data in application
I did not make it in common nice way for SQLalchemy, since was used with flask-restful, so I have only adoption for flask-restful in one direction, but I think it's easy to get the idea from here and do the rest.
And if your case is simple key-value storage so nothing additionaly should be done, just use it as is.
But if you want to unwrap JSON somewhere in code, it's simple like this whenever you use it, just wrap in function
if (value.startswith("{") and value.endswith("}")) or (value.startswith("[") and value.endswith("]")):
value=json.loads(value)
Creating dynamical field for actual nice non-JSON way for editing of data also possible by extending FormField and adding some javascript for adding/removing fields, but this is whole different story, in my case I needed actual json storage, with blackjack and lists :)
Was working on postgres JSON datatype. The above solution worked great with a minor modifications.
Tried
'sqlalchemy.dialects.postgresql.json.JSON',
'sqlalchemy.dialects.postgresql.JSON',
'dialects.postgresql.json.JSON',
'dialects.postgresql.JSON'
The above versions did not work.
Finally the following change worked
#converts('JSON')
And changed class DictToHstoreField to the following:
class DictToJSONField(fields.TextAreaField):
def process_data(self, value):
if value is None:
value = {}
self.data = json.dumps(value)
def process_formdata(self, valuelist):
if valuelist:
self.data = json.loads(valuelist[0])
else:
self.data = '{}'
Although, this is might not be the answer to your question, but by default SQLAlchemy's ORM doesn't detect in-place changes to HSTORE field values. But fortunately there's a solution: SQLAlchemy's MutableDict type:
from sqlalchemy.ext.mutable import MutableDict
class MyClass(Base):
__tablename__ = 'mytable'
id = Column(Integer, primary_key=True)
attributes = Column(MutableDict.as_mutable(HSTORE))
Now when you change something in-place:
my_object.attributes.['some_key'] = 'some value'
The hstore field will be updated after session.commit().