I would like to generate a simple json file from an database.
I am not an expert in parsing json files using python nor NDB database engine nor GQL.
What is the right query to search the data? see https://developers.google.com/appengine/docs/python/ndb/queries
How should I write the code to generate the JSON using the same schema as the json described here below?
Many thanks for your help
Model Class definition using NDB:
# coding=UTF-8
from google.appengine.ext import ndb
import logging
class Albums(ndb.Model):
"""Models an individual Event entry with content and date."""
SingerName = ndb.StringProperty()
albumName = ndb.StringProperty()
Expected output:
{
"Madonna": ["Madonna Album", "Like a Virgin", "True Blue", "Like a Prayer"],
"Lady Gaga": ["The Fame", "Born This Way"],
"Bruce Dickinson": ["Iron Maiden", "Killers", "The Number of the Beast", "Piece of Mind"]
}
For consistency, model names should by singular (Album not Albums), and property names should be lowercase_with_underscores:
class Album(ndb.Model):
singer_name = ndb.StringProperty()
album_name = ndb.StringProperty()
To generate the JSON as described in your question:
1) Query the Album entities from the datastore:
albums = Album.query().fetch(100)
2) Iterate over them to form a python data structure:
albums_dict = {}
for album in albums:
if not album.singer_name in albums_dict:
albums_dict[album.singer_name] = []
albums_dict[album.singer_name].append(album.album_name)
3) use json.dumps() method to encode to JSON.
albums_json = json.dumps(albums_dict)
Alternatively, you could use the built in to_dict() method:
albums = Album.query().fetch(100)
albums_json = json.dumps([a.to_dict() for a in albums])
Related
I am new to django and I am working on project where I am storing a key value pair dictionary as JSON in database.
Now later I want to show it on the html page all the list of packages but not able to access those key value pair as dictionary.
here is my
models.py
class Packages(models.Model):
package_ID = models.AutoField("Package ID", primary_key=True)
package_Name = models.CharField("Package Name", max_length=30, null=False)
attribute_values = models.CharField("Item Details JSON", max_length=500, null=False)
package_Price = models.IntegerField("Package Price", null=False, default=0.00)
quantity = models.IntegerField("Quantity", null=False, default=00)
prod_ID = models.ForeignKey(Product, on_delete=models.CASCADE, verbose_name="Product ID (FK)")
its data entry is something like this
package_ID = 1
package_Name = "basic card"
attribute_values = {"sizes": "8.5 in. x 11 in.", "Colour": "Full-Color Front - Unprinted Back",}
package_Price = 200
quantity = 400
prod_ID = 1
package_ID = 2
package_Name = "best card"
attribute_values = {"sizes": "8.5 in. x 11 in.", "Colour": "Full-Color Front - Unprinted Back",}
package_Price = 200
quantity = 500
prod_ID = 1
here the problem is that attribute_values fields stores a JSON string so I have to convert it in dictionary first and then have to access its dictionary.
the steps I want to do:
get all the packages having same product key:
now get the attribute_values of those packages
and convert that JSON into dictionary
then display the attribute_values of each package separately in html in particular block.
I want output something like this:
but I am getting this:
With Django version >=3.1 one can simply use JSONField for storing JSON in databases.
It is supported for the following databases:
JSONField is supported on MariaDB 10.2.7+, MySQL 5.7.8+, Oracle,
PostgreSQL, and SQLite 3.9.0+ (with the JSON1 extension
enabled).
Using the JSONField should be pretty simple:
class Packages(models.Model):
# other fields
attribute_values = models.JSONField()
Let's say we have an instance of Packages we can simply set a dictionary / variable that can be valid JSON on to attribute_values:
package.attribute_values = {"sizes": "8.5 in. x 11 in.", "Colour": "Full-Color Front - Unprinted Back",}
package.save()
This will be stored as JSON in the database.
When we want to access this in python it would automatically be deserialized to their python value:
package.attribute_values['new_key'] = 'new value'
package.save()
Considering your current problem with the text field you need to simply convert your string to equivalent python objects. This can be done using json.loads:
import json
attribute_values_dictionary = json.loads(package.attribute_values)
You can also add a method to your model to do this for you:
import json
class Packages(models.Model):
# Your fields
def attribute_values_to_python(self):
return json.loads(self.attribute_values)
Now in the view you can simply write:
attribute_values_dictionary = package.attribute_values_to_python()
In the template you can simply write:
{{ package.attribute_values_to_python }}
i tried to fetch data from mongodb using mongoengine with flask. query is work perfect the problem is when i convert query result into json its show only fields name.
here is my code
view.py
from model import Users
result = Users.objects()
print(dumps(result))
model.py
class Users(DynamicDocument):
meta = {'collection' : 'users'}
user_name = StringField()
phone = StringField()
output
[["id", "user_name", "phone"], ["id", "user_name", "phone"]]
why its show only fields name ?
Your query returns a queryset. Use the .to_json() method to convert it.
Depending on what you need from there, you may want to use something like json.loads() to get a python dictionary.
For example:
from model import Users
# This returns <class 'mongoengine.queryset.queryset.QuerySet'>
q_set = Users.objects()
json_data = q_set.to_json()
# You might also find it useful to create python dictionaries
import json
dicts = json.loads(json_data)
I have a MySQL database with a table containing a JSON field called things. The JSON looks like this
things = {"value1": "phil", "value2": "jill"}
I have collection of objects that I have pulled from the database via
my_things = Name_table.objects.values
Now, I'd like to filter the my_things collection by one of the JSON fields. I've tried this
my_things =
my_things.filter(things__contains={'value': 'phil'})
which returned an empty collection. I've also tried
my_things = my_things.filter(things={'value': 'phil'})
and
my_things = my_things.filter(things__exact={'value': 'phil'})
I'n using Django 1.10 and MySQL 5.7
Thoughts?
It depends on how exactly do you store JSON in field. If you use django-jsonfield, then your things will be string without spaces, with strings inside of quotation marks: '{"value1":"phil","value2":"jill"}'.
Then, via docs:
my_things = my_things.filter(things__contains='"value1":"phil"')
should return your filtered QuerySet, because
>>> tmp_str = '{"value1":"phil","value2":"jill"}'
>>> '"value1":"phil"' in tmp_str
True
I have a function which returns json data as history from Version of reversion.models.
from django.http import HttpResponse
from reversion.models import Version
from django.contrib.admin.models import LogEntry
import json
def history_list(request):
history_list = Version.objects.all().order_by('-revision__date_created')
data = []
for i in history_list:
data.append({
'date_time': str(i.revision.date_created),
'user': str(i.revision.user),
'object': i.object_repr,
'field': i.revision.comment.split(' ')[-1],
'new_value_field': str(i.field_dict),
'type': i.content_type.name,
'comment': i.revision.comment
})
data_ser = json.dumps(data)
return HttpResponse(data_ser, content_type="application/json")
When I run the above snippet I get the output json as
[{"type": "fruits", "field": "colour", "object": "anyobject", "user": "anyuser", "new_value_field": "{'price': $23, 'weight': 2kgs, 'colour': 'red'}", "comment": "Changed colour."}]
From the function above,
'comment': i.revision.comment
returns json as "comment": "changed colour" and colour is the field which I have written in the function to retrieve it from comment as
'field': i.revision.comment.split(' ')[-1]
But i assume getting fieldname and value from field_dict is a better approach
Problem: from the above json list I would like to filter new_field_value and old_value. In the new_filed_value only value of colour.
Getting the changed fields isn't as easy as checking the comment, as this can be overridden.
Django-reversion just takes care of storing each version, not comparing.
Your best option is to look at the django-reversion-compare module and its admin.py code.
The majority of the code in there is designed to produce a neat side-by-side HTML diff page, but the code should be able to be re-purposed to generate a list of changed fields per object (as there can be more than one changed field per version).
The code should* include a view independent way to get the changed fields at some point, but this should get you started:
from reversion_compare.admin import CompareObjects
from reversion.revisions import default_revision_manager
def changed_fields(obj, version1, version2):
"""
Create a generic html diff from the obj between version1 and version2:
A diff of every changes field values.
This method should be overwritten, to create a nice diff view
coordinated with the model.
"""
diff = []
# Create a list of all normal fields and append many-to-many fields
fields = [field for field in obj._meta.fields]
concrete_model = obj._meta.concrete_model
fields += concrete_model._meta.many_to_many
# This gathers the related reverse ForeignKey fields, so we can do ManyToOne compares
reverse_fields = []
# From: http://stackoverflow.com/questions/19512187/django-list-all-reverse-relations-of-a-model
changed_fields = []
for field_name in obj._meta.get_all_field_names():
f = getattr(
obj._meta.get_field_by_name(field_name)[0],
'field',
None
)
if isinstance(f, models.ForeignKey) and f not in fields:
reverse_fields.append(f.rel)
fields += reverse_fields
for field in fields:
try:
field_name = field.name
except:
# is a reverse FK field
field_name = field.field_name
is_reversed = field in reverse_fields
obj_compare = CompareObjects(field, field_name, obj, version1, version2, default_revision_manager, is_reversed)
if obj_compare.changed():
changed_fields.append(field)
return changed_fields
This can then be called like so:
changed_fields(MyModel,history_list_item1, history_list_item2)
Where history_list_item1 and history_list_item2 correspond to various actual Version items.
*: Said as a contributor, I'll get right on it.
I'm a new spark user currently playing around with Spark and some big data and I have a question related to Spark SQL or more formally the SchemaRDD. I'm reading a JSON file containing data about some weather forecasts and I'm not really interested in all of the fields that I have ... I only want 10 fields out of 50+ fields returned for each record. Is there a way (similar to filter) that I can use to specify the names of some fields that I want remove from spark.
Just a small descriptive example. Consider I have the Schema "Person" with 3 fields "Name", "Age", and "Gender" and I'm not interested in the "Age" field and wold like to remove it. Can I use spark some how to do that. ? Thanks
If you are using Spark 1.2, you can do the following (using Scala)...
If you already know what fields you want to use, you can construct the schema for these fields and apply this schema to the JSON dataset. Spark SQL will return a SchemaRDD. Then, you can register it and query it as a table. Here is a snippet...
// sc is an existing SparkContext.
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
// The schema is encoded in a string
val schemaString = "name gender"
// Import Spark SQL data types.
import org.apache.spark.sql._
// Generate the schema based on the string of schema
val schema =
StructType(
schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
// Create the SchemaRDD for your JSON file "people" (every line of this file is a JSON object).
val peopleSchemaRDD = sqlContext.jsonFile("people.txt", schema)
// Check the schema of peopleSchemaRDD
peopleSchemaRDD.printSchema()
// Register peopleSchemaRDD as a table called "people"
peopleSchemaRDD.registerTempTable("people")
// Only values of name and gender fields will be in the results.
val results = sqlContext.sql("SELECT * FROM people")
When you look at the schema of peopleSchemaRDD (peopleSchemaRDD.printSchema()), you will only see name and gender field.
Or, if you want to explore the dataset and determine what fields you want after you see all fields, you can ask Spark SQL to infer the schema for you. Then, you can register the SchemaRDD as a table and use projection to remove unneeded fields. Here is a snippet...
// Spark SQL will infer the schema of the given JSON file.
val peopleSchemaRDD = sqlContext.jsonFile("people.txt")
// Check the schema of peopleSchemaRDD
peopleSchemaRDD.printSchema()
// Register peopleSchemaRDD as a table called "people"
peopleSchemaRDD.registerTempTable("people")
// Project name and gender field.
sqlContext.sql("SELECT name, gender FROM people")
You can specify what fields you would like to have in the schemaRDD. Below is an example. Create a case class, with only the fields that you need. Read the data into an rdd, then specify the only the fileds that you need(in the same order as you have specified the schema in the case class).
Sample Data: People.txt
foo,25,M
bar,24,F
Code:
case class Person(name: String, gender: String)
val people = sc.textFile("People.txt").map(_.split(",")).map(p => Person(p(0), p(2)))
people.registerTempTable("people")