how do I autoconvert a struct* to a dict? - cython

mystruct.pxd
ctypedef struct foo:
pass
ctypedef struct myStruct:
int field1
int field2
foo* field3
mystruct.pyx
class MyStruct:
cdef myStruct* _ptr
def __cinit__(self):
self._ptr = create_myStruct()
def __getattr__(self, key):
if key in self._ptr[0]:
return self._ptr[0][key]
__getattr__ does not compile with cython. Attempting to index non-array type 'myStruct'
Removing the [0] results in a compiler crash.
Attempts to do:
def get_dict(self):
return self._ptr # cannot convert 'myStruct *' to Python object
return self._ptr[0] # cannot convert 'myStruct' to Python object
These structs have many fields that I need to access as properties. Writing basic #property wrappers is adding a lot of clutter to the file so I'd like to be able to use a simple getaddr instead. Any thought?
UPDATE: changed type of field3

It seems the problem is with foo* field3 since foo is a custom struct and not a primitive type (numeric, char*) autoconversion will not work. There does not appear to be a way to fix this directly. I have two workarounds:
Change the type of field3 to unsigned int (or whatever your pointer size is) and cast to foo* when you need to access the sub-structure. This is not a very nice workaround but if your members are really opaque types and not accessible structs, it works just fine.
Write something to explicitly convert to a dict.
This is what I ended up doing. It's another repetition of the field names but it can be easily created with copy-paste-modify
def to_dict(self):
return {'field1':self._ptr.field1,
'field2':self._ptr.field2,
'field3':FooWrapper().from_ptr(self._ptr.field3) # or an int if opaque
}
cdef class FooWrapper:
cdef foo* _ptr
def from_ptr(self, ptr):
assert(ptr is not NULL)
self._ptr = ptr
return self

Related

using special function such as __add__ in cython cdef classes

I am wanting to create a cython object that can has convenient operations such as addition multiplication and comparisons. But when I compile such classes they all seem to have a lot of python overhead.
A simple example:
%%cython -a
cdef class Pair:
cdef public:
int a
int b
def __init__(self, int a, int b):
self.a = a
self.b = b
def __add__(self, Pair other):
return Pair(self.a + other.a, self.b + other.b)
p1 = Pair(1, 2)
p2 = Pair(3, 4)
p3 = p1+p2
print(p3.a, p3.b)
But I end up getting quite large readouts from the annotated compiler
It seems like the __add__ function is converting objects from python floats to cython doubles and doing a bunch of type checking. Am I doing something wrong?
There's likely a couple of issues:
I'm assuming that you're using Cython 0.29.x (and not the newer Cython 3 alpha). See https://cython.readthedocs.io/en/stable/src/userguide/special_methods.html#arithmetic-methods
This means that you can’t rely on the first parameter of these methods being “self” or being the right type, and you should test the types of both operands before deciding what to do
It is likely treating self as untyped and thus accessing a and b as Python attributes.
The Cython 3 alpha treats special methods differently (see https://cython.readthedocs.io/en/latest/src/userguide/special_methods.html#arithmetic-methods) so you could also consider upgrading to that.
Although the call to __init__ has C typed arguements it's still a Python call so you can't avoid boxing and unboxing the arguments to Python ints. You could avoid this call and do something like:
cdef Pair res = Pair.__new__()
res.a = ... # direct assignment to attribute

Super constructor in cdef classes

I am working on wrapping a C library using cython. I would like to use a cdef class to wrap a simple struct. The struct does require some parameters to be properly initialized. I put the corresponding code int the __cinit__ method like so:
cdef class Func:
cdef library.Func* func
def __cinit__(self, int value):
library.init_func(&self.func, value)
In python code I can create a function object via f = Func(17). The code then handles the initialization just fine. My question is the following: Say I want to extend the Func class in python (class MyFunc(Func): ...). How do I add a constructor to to MyFunc?
I would like to write a constructor with other parameters which calls
__cinit__(self, int value) with a value derived from constructor parameters. But it does not seem to be possible (likely for good reason) to call __cinit__ from python code.
Is there some way to include constructors in subclasses or should I create a wrapper around the function and delegate methods / properties?

Python Function Inputs with Varying Data Types

I'm brand new to Python and come from a C/C++ background. I'm looking for a way to differentiate function calls based on the data that is being passed.
An example:
class RomanNumeral():
def __init__( self, value ):
# Assign stuff
Now, let's say I'd like to be able to initialize an instance of RomanNumeral with either an int or a string (e.g., 11 or XI). How is this handled in Python?
I'm essentially looking for this C++ equivalent:
RomanNumeral();
RomanNumeral(int value);
RomanNumeral(string value);
Do I have to throw a try block around some code and catch ValueError if I'm given a string?
Python does not complain about type mismatch like C++. Syntactically, you can assign any data type to any variable. For your case,
class RomanNumeral:
def __init__(self, value):
self.value = value
And in any method you use it, you can:
def some_method(self):
if isinstance(self.value, int):
print "Value is a number"
elif isinstance(self.value, str):
print "Value is a Roman Number"
This is a simple way to handle types in Python

How to decorate an immutable object graph from scala case classes

I'm reading structured JSON, using Play Frameworks' JSON Reads to build up an object graph with case classes.
An example:
case class Foo (
id: Int,
bar_id: Int,
baz_id: Int,
x: Int,
y: String
)
{
var bar: Bar = null
var baz: Baz = null
}
After building the Foo, I must come back later and decorate it by setting bar and baz. Those are defined in other JSON files and only known when all parsing is complete. But this means Foo can't be immutable.
What is the "right" way in Scala to make an immutable object, and then a decorated version of it, without repeating every field of Foo multiple times, over and over?
I know several ways that feel wrong:
make "bar: Option[Bar]" and "baz: Option[Baz]" case class parameters, and then I can use "copy" to make new versions of the Foo class with them set to something; but then I have to check them every single time they're accessed - inefficient, unsafe, not able to make a DecoratedFoo that just is guaranteed to have the right structure
make a second case class that is a copy-paste of all the structure in the first, but adding the two extra decorated parameters - but this means echoing the entire parameter list in the definition, and again when creating instances of it
Case class inheritance is apparently controversial, and in any case also appears to require me to repeat every single parameter anyway, in the subclass constructor?
Make a non-case superclass listing the common case class parameters. Then extend it in the case class. But this would seem to still require repeating every single parameter in the subclass constructor as well.
I see blogs with people talking about this problem and using reflection at runtime to populate the base attributes of their decorated copies - this avoids echo but now you have no type safety, specifying attribute names as strings, overhead, etc...
Surely Scala must have a way to let people compose more complicated immutable objects out of simpler ones without having to copy each and every part of them by hand?
You could introduce a new trait for the processed types, a class that extends that trait, and an implicit conversion:
case class Foo(bar: Int)
trait HasBaz {
val baz: Int
}
class FooWithBaz(val foo: Foo, val baz: Int) extends HasBaz
object FooWithBaz {
implicit def innerFoo(fwb: FooWithBaz): Foo = fwb.foo
implicit class RichFoo(val foo: Foo) extends AnyVal {
def withBaz(baz: Int) = new FooWithBaz(foo, baz)
}
}
So then you can do:
import FooWithBaz._
Foo(1).withBaz(5)
And, although withBaz returns a FooWithBaz, we can treat the return value like a Foo when necessary, because of the implicit conversion.
Combining Option and type parameters you can flag your case class, and track whether the processed fields are empty, statically:
import scala.language.higherKinds
object Acme {
case class Foo[T[X] <: Option[X] forSome { type X }](a: Int,
b: String,
c: T[Boolean],
d: T[Double])
// Necessary, Foo[None] won't compile
type Unprocessed[_] = None.type
// Just an alias
type Processed[X] = Some[X]
}
Example use case:
import Acme._
val raw: Foo[Unprocessed] = Foo[Unprocessed](42, "b", None, None)
def process(unprocessed: Foo[Unprocessed]): Foo[Processed] =
unprocessed.copy[Processed](c = Some(true), d = Some(42d))
val processed: Foo[Processed] = process(raw)
// No need to pattern match, use directly the x from the Some case class
println(processed.c.x)
println(processed.d.x)
I used this once in my current project. The main problem I encountered is when I want Foo to be covariant.
Alternatively, if you don't care about the bound on T:
case class Foo[+T[_]](a: Int, b: String, c: T[Boolean], d: T[Double])
then you can use Foo[Unprocessed] or Foo[Processed] when you need a Foo[Option].
scala> val foo: Foo[Option] = processed
foo: Acme.Foo[Option] = Foo(42,b,Some(true),Some(42.0))
One other strategy might be to create yet another case class:
case class Foo(
id: Int,
bar_id: Int,
baz_id: Int,
x: Int,
y: String
)
case class ProcessedFoo(
foo: Foo,
bar: Bar,
baz: Baz
)

getting struct elements in Cython

Amazingly I can't seem to find a single example of getting elements of a struct, by name (both on the web and in the cython examples).
So I'm receiving a pointer to a struct out of a C function, and would want to access those elements one by one and repackage them into a python list/dict.
maybe:
structPointer['propertyName']
or
structPointer.propertyName
I want to get the effect of structName->propertyName.
Your second syntax is the correct one, but you have to have an extern declaration for the struct type:
cdef extern from "someheader.h":
struct properties_t:
int value1
int value2
properties_t* getthem()
cdef void foo():
cdef properties_t* prop
prop = getthem()
i = prop.value1