Related
I know I can use ruamel.yaml to load a file with tags in it. But when I want to dump without them i get an error. Simplified example :-
from ruamel.yaml import YAML
from json import dumps
import sys
yaml = YAML()
data = yaml.load(
"""
!mytag
a: 1
b: 2
c: 2022-05-01
"""
)
try:
yaml2 = YAML(typ='safe', pure=True)
yaml.default_flow_style = True
yaml2.dump(data, sys.stdout)
except Exception as e:
print('exception dumping using yaml', e)
try:
print(dumps(data))
except Exception as e:
print('exception dumping using json', e)
exception dumping using cannot represent an object: ordereddict([('a', 1), ('b', 2), ('c', datetime.date(2022, 5, 1))])
exception dumping using json Object of type date is not JSON serializable
I cannot change the load() without getting an error on the tag. How to get output with tags stripped (YAML or JSON)?
You get the error because the neither the safe dumper (pure or not), nor JSON, do know about the ruamel.yaml internal
types that preserve comments, tagging, block/flow-style, etc.
Dumping as YAML, you could register these types with alternate dump methods. As JSON this is more complex
as AFAIK you can only convert the leaf-nodes (i.e. the YAML scalars, you would e.g. be
able to use that to dump the datetime.datetime instance that is loaded as the value of key c).
I have used YAML as a readable, editable and programmatically updatable config file with
an much faster loading JSON version of the data used if its file is not older than the corresponding YAML (if
it is older JSON gets created from the YAML). The thing to do in order to dump(s) is
recursively generate Python primitives that JSON understands.
The following does so, but there are other constructs besides datetime
instances that JSON doesn't allow. E.g. when using sequences or dicts
as keys (which is allowed in YAML, but not in JSON). For keys that are
sequences I concatenate the string representation of the elements
:
from ruamel.yaml import YAML
import sys
import datetime
import json
from collections.abc import Mapping
yaml = YAML()
data = yaml.load("""\
!mytag
a: 1
b: 2
c: 2022-05-01
[d, e]: !myseq [42, 196]
{f: g, 18: y}: !myscalar x
""")
def json_dump(data, out, indent=None):
def scalar(obj):
if obj is None:
return None
if isinstance(obj, (datetime.date, datetime.datetime)):
return str(obj)
if isinstance(obj, ruamel.yaml.scalarbool.ScalarBoolean):
return obj == 1
if isinstance(obj, bool):
return bool(obj)
if isinstance(obj, int):
return int(obj)
if isinstance(obj, float):
return float(obj)
if isinstance(obj, tuple):
return '_'.join([str(x) for x in obj])
if isinstance(obj, Mapping):
return '_'.join([f'{k}-{v}' for k, v in obj.items()])
if not isinstance(obj, str): print('type', type(obj))
return obj
def prep(obj):
if isinstance(obj, dict):
return {scalar(k): prep(v) for k, v in obj.items()}
if isinstance(obj, list):
return [prep(elem) for elem in obj]
if isinstance(obj, ruamel.yaml.comments.TaggedScalar):
return prep(obj.value)
return scalar(obj)
res = prep(data)
json.dump(res, out, indent=indent)
json_dump(data, sys.stdout, indent=2)
which gives:
{
"a": 1,
"b": 2,
"c": "2022-05-01",
"d_e": [
42,
196
],
"f-g_18-y": "x"
}
Currently working on a quick little project in python and am attempting to encode an object into a JSON string. I've done this several times before without any problem except for now. Usually I just do the following.
def ClassToEncode :
def __init__(self, arg1, arg2, ..., argn) :
self.attr1 = arg1
self.attr2 = arg2
...
self.attrn = argn
...
def toJSON(self) :
return json.dumps(self, default=lambda o: o.__dict__)
But the problem is that one of my class attributes is a datetime.datetime object and I am being thrown the following error
AttributeError: 'datetime.datetime' object has no attribute '__dict__'
Any thoughts or wraparounds that could enable the functionality of including the datetime attribute into the JSON output??
Thanks in advance!
You can use the isoformat() method on a datetime object to convert it to an ISO-8601-formatted time string, then serialize it as JSON just fine. On the other end, call datetime.datetime.strptime() on the formatted string to convert it back into a datetime object:
>>> from datetime import datetime as dt
>>> now = dt.now()
>>> now
datetime.datetime(2014, 9, 4, 3, 19, 44, 214096)
>>> isonow = now.isoformat()
>>> isonow
'2014-09-04T03:19:44.214096'
>>> fmt = "%Y-%m-%dT%H:%M:%S.%f"
>>> newtime = dt.strptime(isonow, fmt)
>>> newtime
datetime.datetime(2014, 9, 4, 3, 19, 44, 214096)
Another way is to modify your toJSON() method to use a customized dictionary in which you customize the data:
import datetime
def ClassToEncode :
def __init__(self, arg1, arg2, ..., argn) :
self.attr1 = arg1
self.attr2 = arg2
...
self.attrn = datetime.datetime.utcnow()
...
def customDict(self):
dup = self.__dict__.copy()
# configure dup to contain fields that you want to send
dup['attrn'] = self.createdAt.isoformat() # datetime object
del dup['attr2'] # Some private field you may want to hide
return dup
def toJSON(self):
return json.dumps(self, default=lambda o: o.customDict())
While the accepted answer works great. If you don't need any date formatting you could quickly get away with direct string conversion.
So something like below should work
str(datetime.datetime.now())
json.dumps should be able to serialize this string now.
according to http://dev.mysql.com/doc/refman/5.0/en/datetime.html. i got to find a way to convert the string value 'YYYY-MM-DD HH:MM:SS' to a timestamp int.
i looked up in python's doc.
i tried:
print(time.strptime('2013-01-12 15:27:43', '%Y-%m-%d %H:%M:%S'))
python give me a result like this.
time.struct_time(tm_year=2013, tm_mon=1, tm_mday=12, tm_hour=15, tm_min=27, tm_sec=43, tm_wday=5, tm_yday=12, tm_isdst=-1)
i tried this to convert timestamp to YYYY-MM-DD HH:MM:SS format
print(time.strftime('%Y-%m-%d %H:%M:%S',time.time()))
python give me a type error.
i only use timestamp to calculate time and date, i hope there's already a way in python, simple and efficient , and don't have to create temp data.
according to the answer i write two methods. hope it would be helpful
import time
def convertTimestampToSQLDateTime(value):
return time.strftime('%Y-%m-%d %H:%M:%S',time.localtime(value))
def convertSQLDateTimeToTimestamp(value):
return time.mktime(time.strptime(value, '%Y-%m-%d %H:%M:%S'))
Happy to update this if I'm not properly understanding, but here are a few examples which may help. Note that this uses the datetime module instead of time.
>>> import datetime
Here we set up an example timestamp ts and a format f:
>>> ts = '2013-01-12 15:27:43'
>>> f = '%Y-%m-%d %H:%M:%S'
Similar to what you did above, we use the strptime function (from datetime.datetime) to convert our string into a datetime object based on the formatting parameter:
>>> datetime.datetime.strptime(ts, f)
datetime.datetime(2013, 1, 12, 15, 27, 43)
Now in reverse - here we use datetime.datetime.now() to get the current time as a datetime object:
>>> now = datetime.datetime.now()
>>> now
datetime.datetime(2013, 1, 12, 0, 46, 54, 490219)
In the datetime case, the strftime method is actually called on the datetime object itself, with the formatting parameter as an argument:
>>> now.strftime(f)
'2013-01-12 00:46:54'
In your situation, the reason you were getting an error is because time.time() returns a float:
>>> time.time()
1357980846.290231
But time.strftime needs a time tuple, similar to what you had above. Without getting into the maddening spiral that is time, a function such as time.localtime() will return the aforementioned time tuple and will return as you expect:
>>> now = time.localtime()
>>> now
time.struct_time(tm_year=2013, tm_mon=1, tm_mday=12, tm_hour=0, tm_min=55, tm_sec=55, tm_wday=5, tm_yday=12, tm_isdst=0)
>>> f = '%Y-%m-%d %H:%M:%S'
>>> time.strftime(f, now)
'2013-01-12 00:55:55'
I'm only adding this class to potentially save the next guy a little time. If anyone finds this useful, upvote RocketDonkey's answer.
## dev on v3.7.6
from datetime import datetime
from time import mktime, time
class Time:
'''\
*Convenience class for easy format conversion*\n
Accepts time() float, datetime object, or SQL datetime str.\n
If no time arg is provided, object is initialized with time().\n
id kwarg can be used to keep track of objects.\n
Access formats as instance.t, instance.dt, or instance.sql.\
'''
f = '%Y-%m-%d %H:%M:%S'
def __init__(self, *arg, id=None) -> None:
self.id = id
if len(arg) == 0:
self.t = time()
self.dt = self._dt
self.sql = self._sql
else:
arg = arg[0]
if isinstance(arg, float) or arg == None:
if isinstance(arg, float):
self.t = arg
else:
self.t = time()
self.dt = self._dt
self.sql = self._sql
elif isinstance(arg, datetime):
self.t = arg.timestamp()
self.dt = arg
self.sql = self._sql
elif isinstance(arg, str):
self.sql = arg
if '.' not in arg:
self.dt = datetime.strptime(self.sql, Time.f)
else:
normal, fract = arg.split('.')
py_t = datetime.strptime(normal, Time.f)
self.dt = py_t.replace(
microsecond=int(fract.ljust(6, '0')[:6]))
self.t = self.dt.timestamp()
#property
def _dt(self) -> datetime:
return datetime.fromtimestamp(self.t)
#property
def _sql(self) -> str:
t = self.dt
std = t.strftime(Time.f)
fract = f'.{str(round(t.microsecond, -3))[:3]}'
return std + fract
def __str__(self) -> str:
if self.id == None:
return self.sql
else:
return f'Time obj "{self.id}": {self.sql}'
def test():
def test_one(*arg):
t = Time(*arg, id=type(*arg))
print(t)
print(t.t)
print(t.dt)
sql = '2020-01-22 15:30:33.433'
time_float = 1579927395.3708763
dt_obj = datetime.now()
for datum in [sql, time_float, dt_obj, None]:
test_one(datum)
if __name__ == '__main__':
test()
I'm working with nested JSON-like data structures in python 2.7 that I exchange with some foreign perl code. I just want to 'work with' these nested structures of lists and dictionaries in amore pythonic way.
So if I have a structure like this...
a = {
'x': 4,
'y': [2, 3, { 'a': 55, 'b': 66 }],
}
...I want to be able to deal with it in a python script as if it was nested python classes/Structs, like this:
>>> aa = j2p(a) # <<- this is what I'm after.
>>> print aa.x
4
>>> aa.z = 99
>>> print a
{
'x': 4,
'y': [2, 3, { 'a': 55, 'b': 66 }],
'z': 99
}
>>> aa.y[2].b = 999
>>> print a
{
'x': 4,
'y': [2, 3, { 'a': 55, 'b': 999 }],
'z': 99
}
Thus aa is a proxy into the original structure. This is what I came up with so far, inspired by the excellent What is a metaclass in Python? question.
def j2p(x):
"""j2p creates a pythonic interface to nested arrays and
dictionaries, as returned by json readers.
>>> a = { 'x':[5,8], 'y':5}
>>> aa = j2p(a)
>>> aa.y=7
>>> print a
{'x': [5, 8], 'y':7}
>>> aa.x[1]=99
>>> print a
{'x': [5, 99], 'y':7}
>>> aa.x[0] = {'g':5, 'h':9}
>>> print a
{'x': [ {'g':5, 'h':9} , 99], 'y':7}
>>> print aa.x[0].g
5
"""
if isinstance(x, list):
return _list_proxy(x)
elif isinstance(x, dict):
return _dict_proxy(x)
else:
return x
class _list_proxy(object):
def __init__(self, proxied_list):
object.__setattr__(self, 'data', proxied_list)
def __getitem__(self, a):
return j2p(object.__getattribute__(self, 'data').__getitem__(a))
def __setitem__(self, a, v):
return object.__getattribute__(self, 'data').__setitem__(a, v)
class _dict_proxy(_list_proxy):
def __init__(self, proxied_dict):
_list_proxy.__init__(self, proxied_dict)
def __getattribute__(self, a):
return j2p(object.__getattribute__(self, 'data').__getitem__(a))
def __setattr__(self, a, v):
return object.__getattribute__(self, 'data').__setitem__(a, v)
def p2j(x):
"""p2j gives back the underlying json-ic json-ic nested
dictionary/list structure of an object or attribute created with
j2p.
"""
if isinstance(x, (_list_proxy, _dict_proxy)):
return object.__getattribute__(x, 'data')
else:
return x
Now I wonder whether there is an elegant way of mapping a whole set of the __*__ special functions, like __iter__, __delitem__? so I don't need to unwrap things using p2j() just to iterate or do other pythonic stuff.
# today:
for i in p2j(aa.y):
print i
# would like to...
for i in aa.y:
print i
I think you're making this more complex than it needs to be. If I understand you correctly, all you should need to do is this:
import json
class Struct(dict):
def __getattr__(self, name):
return self[name]
def __setattr__(self, name, value):
self[name] = value
def __delattr__(self, name):
del self[name]
j = '{"y": [2, 3, {"a": 55, "b": 66}], "x": 4}'
aa = json.loads(j, object_hook=Struct)
for i in aa.y:
print(i)
When you load JSON, the object_hook parameter lets you specify a callable object to process objects that it loads. I've just used it to turn the dict into an object that allows attribute access to its keys. Docs
There is an attrdict library that does exactly that in a very safe manner, but if you want, a quick and dirty (possibly leaking memory) approach was given in this answer:
class AttrDict(dict):
def __init__(self, *args, **kwargs):
super(AttrDict, self).__init__(*args, **kwargs)
self.__dict__ = self
j = '{"y": [2, 3, {"a": 55, "b": 66}], "x": 4}'
aa = json.loads(j, object_hook=AttrDict)
I found the answer: There is intentionally no way to automatically map the special methods in python, using __getattribute__. So to achieve what I want, I need to explicitely define all special methods like __len__ one after the other.
Is there a way to increment the year on filtered objects using the update() method?
I am using:
python 2.6.5
django 1.2.1 final
mysql Ver 14.14 Distrib 5.1.41
I know it's possible to do something like this:
today = datetime.datetime.today()
for event in Event.objects.filter(end_date__lt=today).iterator():
event.start_date = festival.start_date + datetime.timedelta(365)
event.end_date = festival.end_date + datetime.timedelta(365)
event.save()
However, in some cases, I would prefer to use the update() method.
# This does not work..
Event.objects.all().update(
start_date=F('start_date') + datetime.timedelta(365),
end_date=F('end_date') + datetime.timedelta(365)
)
With the example above, I get:
Warning: Truncated incorrect DOUBLE value: '365 0:0:0'
The sql query it's trying to make is:
UPDATE `events_event` SET `start_date` = `events_event`.`start_date` + 365 days, 0:00:00, `end_date` = `events_event`.`end_date` + 365 days, 0:00:00
I found something in the mysql guide, but this is raw sql!
SELECT DATE_ADD('2008-12-15', INTERVAL 1 YEAR);
Any idea?
One potential cause of "Warning: Data truncated for column X" exception is the use of non-whole day values for the timedelta being added to the DateField - it is fine in python, but fails when written to the mysql db. If you have a DateTimeField, it works too, since the precision of the persisted field matches the precision of the timedelta.
I.e.:
>>> from django.db.models import F
>>> from datetime import timedelta
>>> from myapp.models import MyModel
>>> [field for field in MyModel._meta.fields if field.name == 'valid_until'][0]
<django.db.models.fields.DateField object at 0x3d72fd0>
>>> [field for field in MyModel._meta.fields if field.name == 'timestamp'][0]
<django.db.models.fields.DateTimeField object at 0x43756d0>
>>> MyModel.objects.filter(pk=1).update(valid_until=F('valid_until') + timedelta(days=3))
1L
>>> MyModel.objects.filter(pk=1).update(valid_until=F('valid_until') + timedelta(days=3.5))
Traceback (most recent call last):
...
Warning: Data truncated for column 'valid_until' at row 1
>>> MyModel.objects.filter(pk=1).update(timestamp=F('timestamp') + timedelta(days=3.5))
1L
Quick but ugly:
>>> a.created.timetuple()
time.struct_time(tm_year=2000, tm_mon=11, tm_mday=2, tm_hour=2, tm_min=35, tm_se
c=14, tm_wday=3, tm_yday=307, tm_isdst=-1)
>>> time = list(a.created.timetuple())
>>> time[0] = time[0] + 1
>>> time
[2001, 11, 2, 2, 35, 14, 3, 307, -1]
>>>
from dateutil.relativedelta import relativedelta
yourdate = datetime.datetime(2010, 11, 4, 10, 14, 54, 518749)
yourdate += relativedelta(years=+1)
Relativedelta takes many time arguments, from seconds to years...
Have you seen this: Increasing a datetime field with queryset.update ? I also remember successfully using queryset .update() with timedelta with MySQL backend.