I am confused on how to implement a Django model with the value being a special html character code for making a chessboard.
As a reference I'm looking at a sudoku board model:
class Board(models.Model):
name = models.CharField(max_length=4, primary_key=True)
value = models.SmallIntegerField()
The value for sudoku is easy, since the table will only be filled with numbers.
For reference here's a snippet from the sudoku page_data dictionary in views.py giving each table cell its appropriate value:
{"r1c1": 6, "r1c2": 7, "r1c3": 0, ...}
I don't know what to put for my model's value variable:
class Board(models.Model):
name = models.CharField(max_length=2, primary_key=True)
value =
Here's a snippet of where I assign the name/value pairs in my views.py with the special HTML chess piece character codes in my own page_data dictionary:
{"a8": html.unescape('♖'), "b8": html.unescape('♘'), "c8": html.unescape('♗'), ...}
Any help is appreciated.
So you are looking for a way to store your json data in a django model field.
One option would be to convert your json data to a string and store it in a Charfield.
class Board(models.Model):
name = models.CharField(max_length=4, primary_key=True)
value = models.CharField(blank=True, null=True)
Then convert your json data to a string with the following code:
import json
data = {"r1c1": 6, "r1c2": 7, "r1c3": 0}
Board.objects.create(name="bo", value=json.dumps(data))
You can use the json data in the template by using the safe tag:
from django.utils.safestring import SafeString
def view(request):
return render(request, 'template.html', {'board': SafeString(my_board.value)})
If you do not want to convert your json to string, you can take a look at the JSONField.
Related
I am getting an error while working with JSON response:
Error: AttributeError: 'str' object has no attribute 'get'
What could be the issue?
I am also getting the following errors for the rest of the values:
***TypeError: 'builtin_function_or_method' object is not subscriptable
'Phone': value['_source']['primaryPhone'],
KeyError: 'primaryPhone'***
# -*- coding: utf-8 -*-
import scrapy
import json
class MainSpider(scrapy.Spider):
name = 'main'
start_urls = ['https://experts.expcloud.com/api4/std?searchterms=AB&size=216&from=0']
def parse(self, response):
resp = json.loads(response.body)
values = resp['hits']['hits']
for value in values:
yield {
'Full Name': value['_source']['fullName'],
'Phone': value['_source']['primaryPhone'],
"Email": value['_source']['primaryEmail'],
"City": value.get['_source']['city'],
"Zip Code": value.get['_source']['zipcode'],
"Website": value['_source']['websiteURL'],
"Facebook": value['_source']['facebookURL'],
"LinkedIn": value['_source']['LinkedIn_URL'],
"Twitter": value['_source']['Twitter'],
"BIO": value['_source']['Bio']
}
It's nested deeper than what you think it is. That's why you're getting an error.
Code Example
import scrapy
import json
class MainSpider(scrapy.Spider):
name = 'test'
start_urls = ['https://experts.expcloud.com/api4/std?searchterms=AB&size=216&from=0']
def parse(self, response):
resp = json.loads(response.body)
values = resp['hits']['hits']
for value in values:
yield {
'Full Name': value['_source']['fullName'],
'Primary Phone':value['_source']['primaryPhone']
}
Explanation
The resp variable is creating a python dictionary, but there is no resp['hits']['hits']['fullName'] within this JSON data. The data you're looking for, for fullName is actually resp['hits']['hits'][i]['_source']['fullName']. i being an number because resp['hits']['hits'] is a list.
resp['hits'] is a dictionary and therefore the values variable is fine.
But resp['hits']['hits'] is a list, therefore you can't use the get request, and it's only accepts numbers as values within [], not strings. Hence the error.
Tips
Use response.json() instead of json.loads(response.body), since Scrapy v2.2, scrapy now has support for json internally. Behind the scenes it already imports json.
Also check the json data, I used requests for ease and just getting nesting down till I got the data you needed.
Yielding a dictionary is fine for this type of data as it's well structured, but any other data that needs modifying or changing or is wrong in places. Use either Items dictionary or ItemLoader. There's a lot more flexibility in those two ways of yielding an output than yielding a dictionary. I almost never yield a dictionary, the only time is when you have highly structured data.
Updated Code
Looking at the JSON data, there are quite a lot of missing data. This is part of web scraping you will find errors like this. Here we use a try and except block, for when we get a KeyError which means python hasn't been able to recognise the key associated with a value. We have to handle that exception, which we do here by saying to yield a string 'No XXX'
Once you start getting gaps etc it's better to consider an Items dictionary or Itemloaders.
Now it's worth looking at the Scrapy docs about Items. Essentially Scrapy does two things, it extracted data from websites, and it provides a mechanism for storing this data. The way it does this is storing it in a dictionary called Items. The code isn't that much different from yielding a dictionary but Items dictionary allows you to manipulate the extracted data more easily with extra things scrapy can do. You need to edit your items.py first with the fields you want. We create a class called TestItem, we define each field using scrapy.Field(). We then can import this class in our spider script.
items.py
import scrapy
class TestItem(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
full_name = scrapy.Field()
Phone = scrapy.Field()
Email = scrapy.Field()
City = scrapy.Field()
Zip_code = scrapy.Field()
Website = scrapy.Field()
Facebook = scrapy.Field()
Linkedin = scrapy.Field()
Twitter = scrapy.Field()
Bio = scrapy.Field()
Here we're specifying what we want the fields to be, you can't use a string with spaces unfortunately hence why full name is full_name. The field() creates the field of the item dictionary for us.
We import this item dictionary into our spider script with from ..items import TestItem. The from ..items means we're taking the items.py from the parent folder to the spider script and we're importing the class TestItem. That way our spider can populate the items dictionary with our json data.
Note that just before the for loop we instantiate the class TestItem by item = TestItem(). Instantiate means to call upon the class, in this case it makes a dictionary. This means we are creating the item dictionary and then we populate that dictionary with keys and values. You have to does this before you add your keys and values as you can see from within the for loop.
Spider script
import scrapy
import json
from ..items import TestItem
class MainSpider(scrapy.Spider):
name = 'test'
start_urls = ['https://experts.expcloud.com/api4/std?searchterms=AB&size=216&from=0']
def parse(self, response):
resp = json.loads(response.body)
values = response.json()['hits']['hits']
item = TestItem()
for value in values:
try:
item['full_name'] = value['_source']['fullName']
except KeyError:
item['full_name'] = 'No Name'
try:
item['Phone'] = value['_source']['primaryPhone']
except KeyError:
item['Phone'] = 'No Phone number'
try:
item["Email"] = value['_source']['primaryEmail']
except KeyError:
item['Email'] = 'No Email'
try:
item["City"] = value['_source']['activeLocations'][0]['city']
except KeyError:
item['City'] = 'No City'
try:
item["Zip_code"] = value['_source']['activeLocations'][0]['zipcode']
except KeyError:
item['Zip_code'] = 'No Zip code'
try:
item["Website"] = value['AgentMarketingCenter'][0]['Website']
except KeyError:
item['Website'] = 'No Website'
try:
item["Facebook"] = value['_source']['AgentMarketingCenter'][0]['Facebook_URL']
except KeyError:
item['Facebook'] = 'No Facebook'
try:
item["Linkedin"] = value['_source']['AgentMarketingCenter'][0]['LinkedIn_URL']
except KeyError:
item['Linkedin'] = 'No Linkedin'
try:
item["Twitter"] = value['_source']['AgentMarketingCenter'][0]['Twitter']
except KeyError:
item['Twitter'] = 'No Twitter'
try:
item["Bio"]: value['_source']['AgentMarketingCenter'][0]['Bio']
except KeyError:
item['Bio'] = 'No Bio'
yield item
I'm seeking to create my first Django mySQL model that contains a JSON field.
I've read the docs here, but they don't seem to show an example of how to declare a model that contains multiple fields, including JSON fields. I've done a lot of searches on Google and SO, but haven't yet found examples.
What would be the correct way to declare a model that would store data like this?
candy_bar_name='Snickers',
info={
'target_markets': [
{
'market_type': 'type_a',
'included_markets': [array of strings]
},
{
'market_type': 'type_b',
'included_markets': [array of strings]
},
{
'market_type': 'type_c',
'included_markets': [array of strings]
},
],
}
UPDATE 2-13-2020
Responding to comment from #iklinac --
Thanks for this comment. I see this:
from django_mysql.models import JSONField, Model
class ShopItem(Model):
name = models.CharField(max_length=200)
attrs = JSONField()
def __str__(self):
return self.name
...but I'm confused by this:
def __str__(self):
return self.name
How can a model with multiple fields return a single string? I thought possibly this might be a class that defines a single JSON field in a model defined elsewhere. ;)
UPDATE 2-14-2020
This has now been solved via the help of the comments to this post. This works:
from django_mysql.models import JSONField, Model
class CandyBars(models.Model):
candy_bar_name = models.TextField()
info = JSONField()
In my API I have a module, which collects JSON objects obtained via POST request. JSON objects I'm receiving look more or less like this:
{
"id": "bec7426d-c5c3-4341-9df2-a0b2c54ab8da",
"data": {
"temperature": -2.4,
// some other stuff is here as well ...
}
}
The problem is requirement that I have to save both: records from data dictionary and whole data dictionary as a JSONField. My ORM model looks like this:
class Report(BaseModel):
id = models.BigAutoField(primary_key=True)
data = JSONField(verbose_name=_("Data"), encoder=DjangoJSONEncoder, blank=True, null=True)
temperature = models.DecimalField(
max_digits=3,
decimal_places=1,
)
# other properties
Is there any neat way to get both properties in one attempt to deserialize JSON object? Currently I use nested serializers formed like this:
class DataSerializer(serializers.ModelSerializer):
temperature = serializers.DecimalField(
source="temperature", write_only=True, max_digits=3, decimal_places=1
)
class Meta:
model = Report
fields = ("temperature")
class ReportSerializer(serializers.ModelSerializer):
id = serializers.UUIDField(source="uuid", read_only=True)
data = DataSerializer(source="*")
class Meta:
model = Report
fields = ("id", "data")
which obviously does not pass whole data dictionary to validated_data, which means I have to update the JSON field elsewhere (not good since I would like to have both data and temperature at null=False in ORM model). Any good ideas on how to fix this using DRF serializers would be appreciated.
I believe you should be able to override validate method for your serializer where you can "store initial data JSON field" and do the default validation by calling super()... method.
More info https://www.django-rest-framework.org/api-guide/serializers/#validation
Also, there are object-level validation functions available, you can take a look there as well for the initial posted data
https://www.django-rest-framework.org/api-guide/serializers/#object-level-validation
Also, you can override the method run_validation to access the initially passed data object.
I am trying to serialize a model for displaying on an existing front end interface. The model is setup as such:
class Timevalue(models.Model):
time = models.FloatField(blank=True, null=True)
values = JSONField(blank=True, null=True)
The nature of the values is that it has no defined keys, therefore it is using JSON rather than a structured schema. As an end result, I need to RestAPI to output a list of timevalue objects that is flattened so that each element contains the time key as well as all the keys for the values.
So far I have written the following serializer that can return the data in the format of [{'time': 0.01, 'values': {'value1': 1, 'value2': 2, 'value3': 3}}]
class TimevalueSerializer(serializers.Serializer):
time = serializers.FloatField()
values = serializers.JSONField()
However I cannot achieve getting the output in the necessary format: [{'time': 0.01, 'value1': 1, 'value2': 2, 'value3': 3}].
I have tried the following serializer setup:
class TimevaluechildSerializer(serializers.Serializer):
fields = '*'
class TimevalueSerializer(serializers.Serializer):
time = serializers.FloatField()
values = TimevaluechildSerializer('*')
but I cannot work out what to pass to the child serializer in order for it to return all of the key-value pairs.
As this model is used for other views, I prefer to use a Serializer rather than a ModelSerializer.
Hopefully the answer isn't too difficult.
Stu
Maybe using serializer will be hard for this, rather than that, you can send this response manually. For example:
from rest_framework import status
from rest_framework.response import Response
class SomeApiView(ApiView):
resp_list = list()
for i in Timevalues.objects.all():
t = {'time': i.time}
t.update(i.values)
resp_list.append(t)
return Response(resp_list, status=status.HTTP_200_OK)
My Model:
class Person(models.Model):
first_name = models.CharField(max_length=30)
last_name = models.CharField(max_length=30)
phone = models.CharField(max_length=20)
email = models.EmailField()
My View:
def users(request):
people = Person.objects.all()
data = serializers.serialize('json', people)
return JsonResponse(data, safe=False)
All I want back is the data in JSON format. What I'm getting back is this:
"[{\"model\": \"myapp.person\", \"pk\": 1, \"fields\": {\"first_name\": \"ahmet\", \"last_name\": \"arsan\", \"phone\": \"xxx-xxx-xxxx\", \"email\": \"aarsan#xxxxxxxx.com\"}}]"
While technically that is valid JSON, there are 2 problems (for me) with this response:
I don't want those double quotes escaped.
I don't need the model name (myapp.person).
I don't know if/what I'm doing wrong, but it seems like something is off here. Perhaps my query should be returning a dict but I don't know how to get it to do that. I am using Django 1.10.1, Python 3.4.
You already have an answer, but what you're doing wrong is double encoding. JsonResponse serialises to json, but you already have json as that's what's returned from the serialiser.
Either serialise to "python" or use a standard HttpResponse.
I am assuming you are asking this question for an API response. I would suggest using Rest Framework for that as it makes things very easy. You can select your own fields by writing your own serializers for the model.
from rest_framework import serializers
class PersonSerializer(serializers.ModelSerializer):
class Meta:
model = Person
fields = ('first_name', 'last_name', 'phone', 'email')