I am trying to save data in form of JSON (returned as result from POST request)
def get_data(...):
...
try:
_r = requests.post(
_url_list,
headers=_headers
)
return _r.json()
except Exception as ee:
print('Could not get data: {}'.format(ee))
return None
Into a table in SQLITE database as backend.
def add_to_flight_data(_data):
if _data:
try:
new_record = FlightData(data=_data)
db.session.add(new_record)
db.session.commit()
print('Data instertedto DB!')
return "Success"
except Exception as e:
print('Data NOT instertedto DB! {}'.format(e))
pass
This is my simple flask code
import os
import time
import auth
import json
import requests
import datetime
from flask import Flask
from flask_marshmallow import Marshmallow
from flask_sqlalchemy import SQLAlchemy
# from safrs.safrs_types import JSONType
project_dir = os.path.dirname(os.path.abspath(__file__))
database_file = "sqlite:///{}".format(os.path.join(project_dir, "2w.sqlite"))
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = database_file
db = SQLAlchemy(app)
ma = Marshmallow(app)
class FlightData(db.Model):
id = db.Column(db.Integer, primary_key=True)
created = db.Column(db.DateTime, server_default=db.func.now())
json_data = db.Column(db.JSONType, default={})
def __init__(self, data):
self.data = data
It seems like there is perhaps no option to save JSON in sqlite
json_data = db.Column(db.JSONType, default={})
Please ADVISE
Thanks.
I believe that you should be using db.JSON, not db.JSONType as there is no such column type in sqlalchemy.
Regardless of that, SQLite has no JSON data type, so sqlalchemy won't be able to map columns of type db.JSON onto anything. According to the documentation only Postgres and some MySQL are supported. There is support for JSON in SQLite with the JSON1 extension, but sqlalchemy will not be able to make use of it.
Your best bet then is to declare the column as db.Text and use json.dumps() to jsonify the data on write. Alternatively modify your get_data() function to check for a JSON response (check the Content-type header or try calling _r.json() and catching exceptions), and then return _r.content which will already be a JSON string.
Use json.loads() to read data back from the db.
I am having a django app in which I am storing the json variable.I have stored the json variable through admin and I am trying to print it in shell.My main aim is to pass this variable to a webpage with ajax method.But first when I was trying to print it in shell I get this error
__str__ returned non-string (type list)
My models.py is of this form
from django.db import models
from django.utils import timezone
from jsonfield import JSONField
# Create your models here.
class newgrid(models.Model):
data = JSONField()
def __str__(self):
return self.data
My JSON variable is of this form
[{"col":1,"row":1,"size_x":1,"size_y":1},{"col":2,"row":1,"size_x":1,"size_y":1},{"col":3,"row":1,"size_x":1,"size_y":1},{"col":4,"row":1,"size_x":1,"size_y":1},{"col":1,"row":2,"size_x":1,"size_y":1},{"col":2,"row":2,"size_x":1,"size_y":1},{"col":3,"row":2,"size_x":1,"size_y":1},{"col":4,"row":2,"size_x":1,"size_y":1},{"col":1,"row":3,"size_x":1,"size_y":1},{"col":2,"row":3,"size_x":1,"size_y":1},{"col":3,"row":3,"size_x":1,"size_y":1},{"col":4,"row":3,"size_x":1,"size_y":1},{"col":5,"row":1,"size_x":1,"size_y":1}]
In shell I ran following commands
from testforweb.models import newgrid
newgrid.objects.all()
It initially returned this
<QuerySet [<newgrid: newgrid object (1)>]>
But then I added
def __str__(self):
return self.data
Just to see the actual JSON variable.But I am getting the error
How to see the actual JSON variable which I inserted through admin coz I need to send it to webpage as it is
Edit 1
My updated models.py
from django.db import models
from django.utils import timezone
from jsonfield import JSONField
import simplejson as json
# Create your models here.
class newgrid(models.Model):
data = JSONField()
def __str__(self):
json.dumps(self.data)
The __str__ function must return a string:
def __str__(self):
return json.dumps(self.data)
The JSON field will actually decode the JSON into native python types (lists and dictionaries).
The __str___ method is always expected to return a string.
If you want a string representation of the json, you should use json.dumps(self.data) or similar to serialise the data field as the return value of __str__.
Use
def __str__(self):
return '%s' % (self.data)
Instead of
def __str__(self):
return json.dumps(self.data)
I am trying to write a Django Rest Framework API handler that can receive a file as well as a JSON payload. I've set the MultiPartParser as the handler parser.
However, it seems I cannot do both. If I send the payload with the file as a multi part request, the JSON payload is available in a mangled manner in the request.data (first text part until the first colon as the key, the rest is the data). I can send the parameters in standard form parameters just fine - but the rest of my API accepts JSON payloads and I wanted to be consistent. The request.body cannot be read as it raises *** RawPostDataException: You cannot access body after reading from request's data stream
For example, a file and this payload in the request body:
{"title":"Document Title", "description":"Doc Description"}
Becomes:
<QueryDict: {u'fileUpload': [<InMemoryUploadedFile: 20150504_115355.jpg (image/jpeg)>, <InMemoryUploadedFile: Front end lead.doc (application/msword)>], u'{%22title%22': [u'"Document Title", "description":"Doc Description"}']}>
Is there a way to do this? Can I eat my cake, keep it and not gain any weight?
Edit:
It was suggested that this might be a copy of Django REST Framework upload image: "The submitted data was not a file". It is not. The upload and request is done in multipart, and keep in mind the file and upload of it is fine. I can even complete the request with standard form variables. But I want to see if I can get a JSON payload in there instead.
For someone who needs to upload a file and send some data, there is no straight fwd way you can get it to work. There is an open issue in json api specs for this. One possibility i have seen is to use multipart/related as shown here, but i think its very hard to implement it in drf.
Finally what i had implemented was to send the request as formdata. You would send each file as file and all other data as text.
Now for sending the data as text you can have a single key called data and send the whole json as string in value.
Models.py
class Posts(models.Model):
id = models.UUIDField(default=uuid.uuid4, primary_key=True, editable=False)
caption = models.TextField(max_length=1000)
media = models.ImageField(blank=True, default="", upload_to="posts/")
tags = models.ManyToManyField('Tags', related_name='posts')
serializers.py -> no special changes needed, not showing my serializer here as its too lengthy because of the writable ManyToMany Field implimentation.
views.py
class PostsViewset(viewsets.ModelViewSet):
serializer_class = PostsSerializer
parser_classes = (MultipartJsonParser, parsers.JSONParser)
queryset = Posts.objects.all()
lookup_field = 'id'
You will need custom parser as shown below for parsing json.
utils.py
from django.http import QueryDict
import json
from rest_framework import parsers
class MultipartJsonParser(parsers.MultiPartParser):
def parse(self, stream, media_type=None, parser_context=None):
result = super().parse(
stream,
media_type=media_type,
parser_context=parser_context
)
data = {}
# find the data field and parse it
data = json.loads(result.data["data"])
qdict = QueryDict('', mutable=True)
qdict.update(data)
return parsers.DataAndFiles(qdict, result.files)
The request example in postman
EDIT:
see this extended answer if you want to send each data as key value pair
I know this is an old thread, but I just came across this. I had to use MultiPartParser in order to get my file and extra data to come across together. Here's what my code looks like:
# views.py
class FileUploadView(views.APIView):
parser_classes = (MultiPartParser,)
def put(self, request, filename, format=None):
file_obj = request.data['file']
ftype = request.data['ftype']
caption = request.data['caption']
# ...
# do some stuff with uploaded file
# ...
return Response(status=204)
My AngularJS code using ng-file-upload is:
file.upload = Upload.upload({
url: "/api/picture/upload/" + file.name,
data: {
file: file,
ftype: 'final',
caption: 'This is an image caption'
}
});
I send JSON and an image to create/update a product object. Below is a create APIView that works for me.
Serializer
class ProductCreateSerializer(serializers.ModelSerializer):
class Meta:
model = Product
fields = [
"id",
"product_name",
"product_description",
"product_price",
]
def create(self,validated_data):
return Product.objects.create(**validated_data)
View
from rest_framework import generics,status
from rest_framework.parsers import FormParser,MultiPartParser
class ProductCreateAPIView(generics.CreateAPIView):
queryset = Product.objects.all()
serializer_class = ProductCreateSerializer
permission_classes = [IsAdminOrIsSelf,]
parser_classes = (MultiPartParser,FormParser,)
def perform_create(self,serializer,format=None):
owner = self.request.user
if self.request.data.get('image') is not None:
product_image = self.request.data.get('image')
serializer.save(owner=owner,product_image=product_image)
else:
serializer.save(owner=owner)
Example test:
def test_product_creation_with_image(self):
url = reverse('products_create_api')
self.client.login(username='testaccount',password='testaccount')
data = {
"product_name" : "Potatoes",
"product_description" : "Amazing Potatoes",
"image" : open("local-filename.jpg","rb")
}
response = self.client.post(url,data)
self.assertEqual(response.status_code,status.HTTP_201_CREATED)
#Nithin solution works but essentially it means you are sending JSON as strings and hence not using the actual application/json inside the multipart segments.
What we want is to make the backend accept data in the below format
------WebKitFormBoundaryrga771iuUYap8BB2
Content-Disposition: form-data; name="file"; filename="1x1_noexif.jpeg"
Content-Type: image/jpeg
------WebKitFormBoundaryrga771iuUYap8BB2
Content-Disposition: form-data; name="myjson"; filename="blob"
Content-Type: application/json
{"hello":"world"}
------WebKitFormBoundaryrga771iuUYap8BB2
Content-Disposition: form-data; name="isDownscaled"; filename="blob"
Content-Type: application/json
false
------WebKitFormBoundaryrga771iuUYap8BB2--
MultiPartParser works with the above format but will treat those jsons as files. So we simply unmarshal those jsons by putting them to data.
parsers.py
from rest_framework import parsers
class MultiPartJSONParser(parsers.MultiPartParser):
def parse(self, stream, *args, **kwargs):
data = super().parse(stream, *args, **kwargs)
# Any 'File' found having application/json as type will be moved to data
mutable_data = data.data.copy()
unmarshaled_blob_names = []
json_parser = parsers.JSONParser()
for name, blob in data.files.items():
if blob.content_type == 'application/json' and name not in data.data:
mutable_data[name] = json_parser.parse(blob)
unmarshaled_blob_names.append(name)
for name in unmarshaled_blob_names:
del data.files[name]
data.data = mutable_data
return data
settings.py
REST_FRAMEWORK = {
..
'DEFAULT_PARSER_CLASSES': [
..
'myproject.parsers.MultiPartJSONParser',
],
}
This should work now.
The final bit is testing. Since the test client that ships with Django and REST doesn't support multipart JSON, we work around that by wrapping any JSON data.
import io
import json
def JsonBlob(obj):
stringified = json.dumps(obj)
blob = io.StringIO(stringified)
blob.content_type = 'application/json'
return blob
def test_simple(client, png_3x3):
response = client.post(f'http://localhost/files/', {
'file': png_3x3,
'metadata': JsonBlob({'lens': 'Sigma 35mm'}),
}, format='multipart')
assert response.status_code == 200
If you're getting an error along the lines of Incorrect type. Expected pk value, received list., with #nithin's solution, it's because Django's QueryDict is getting in the way - it's specifically structured to use a list for each entry in the dictionary, and thus:
{ "list": [1, 2] }
when parsed by MultipartJsonParser yields
{ 'list': [[1, 2]] }
which trips up your serializer.
Here is an alternative which handles this case, specifically expecting the _data key for your JSON:
from rest_framework import parsers
import json
class MultiPartJSONParser(parsers.MultiPartParser):
def parse(self, stream, *args, **kwargs):
data = super().parse(stream, *args, **kwargs)
json_data_field = data.data.get('_data')
if json_data_field is not None:
parsed = json.loads(json_data_field)
mutable_data = {}
for key, value in parsed.items():
mutable_data[key] = value
mutable_files = {}
for key, value in data.files.items():
if key != '_data':
mutable_files[key] = value
return parsers.DataAndFiles(mutable_data, mutable_files)
json_data_file = data.files.get('_data')
if json_data_file:
parsed = parsers.JSONParser().parse(json_data_file)
mutable_data = {}
for key, value in parsed.items():
mutable_data[key] = value
mutable_files = {}
for key, value in data.files.items():
mutable_files[key] = value
return parsers.DataAndFiles(mutable_data, mutable_files)
return data
It is very simple to use a multipart post and a regular view, if this is an option.
You send the json as a field and files as files, then process in one view.
Here is a simple python client and a Django server:
Client - sending multiple files and an arbitrary json-encoded object:
import json
import requests
payload = {
"field1": 1,
"manifest": "special cakes",
"nested": {"arbitrary":1, "object":[1,2,3]},
"hello": "word" }
filenames = ["file1","file2"]
request_files = {}
url="example.com/upload"
for filename in filenames:
request_files[filename] = open(filename, 'rb')
r = requests.post(url, data={'json':json.dumps(payload)}, files=request_files)
Server - consuming the json and saving the files:
#csrf_exempt
def upload(request):
if request.method == 'POST':
data = json.loads(request.POST['json'])
try:
manifest = data['manifest']
#process the json data
except KeyError:
HttpResponseServerError("Malformed data!")
dir = os.path.join(settings.MEDIA_ROOT, "uploads")
os.makedirs(dir, exist_ok=True)
for file in request.FILES:
path = os.path.join(dir,file)
if not os.path.exists(path):
save_uploaded_file(path, request.FILES[file])
else:
return HttpResponseNotFound()
return HttpResponse("Got json data")
def save_uploaded_file(path,f):
with open(path, 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
I'd just like to add to #Pithikos's answer by modifying the parser to accept lists as well, in line with how DRF parses lists in serializers in utils/html#parse_html_list
class MultiPartJSONParser(parsers.MultiPartParser):
def parse(self, stream, *args, **kwargs):
data = super().parse(stream, *args, **kwargs)
# Any 'File' found having application/json as type will be moved to data
mutable_data = data.data.copy()
unmarshaled_blob_names = []
json_parser = parsers.JSONParser()
for name, blob in data.files.items():
if blob.content_type == 'application/json' and name not in data.data:
parsed = json_parser.parse(blob)
if isinstance(parsed, list):
# need to break it out into [0], [1] etc
for idx, item in enumerate(parsed):
mutable_data[name+f"[{str(idx)}]"] = item
else:
mutable_data[name] = parsed
unmarshaled_blob_names.append(name)
for name in unmarshaled_blob_names:
del data.files[name]
data.data = mutable_data
return data
The following code worked for me.
from django.core.files.uploadedfile import SimpleUploadedFile
import requests
from typing import Dict
with open(file_path, 'rb') as f:
file = SimpleUploadedFile('Your-Name', f.read())
data: Dict[str,str]
files: Dict[str,SimpleUploadedFile] = {'model_field_name': file}
requests.put(url, headers=headers, data=data, files=files)
requests.post(url, headers=headers, data=data, files=files)
'model_field_name' is the name of the FileField or ImageField in your Django model. You can pass other data as name or location as usual by using data parameter.
Hope this helps.
This work for me:
class FileUpload(APIView):
parser_classes = [MultiPartParser]
authentication_classes = [JWTAuthentication]
def post(self, request, filename, format=None):
file = request.data['file']
data = json.loads(request.POST['form'])
#.... just do.....
.
.
.
frontend part: example with fetch (vue frontend)
let data = await new FormData(); // creates a new FormData object
data.append("file", this.files); // add your file to form data
data.append('form',JSON.stringify(body)) //add your json
fetch(`https://endpoint/FileUpload/${body.nombre}`, {
method: 'POST',
body: data,
headers: {Authorization: `Bearer ${accessToken}`}
})
I hope this helps.
I've been trying to update a small Python library called libpynexmo to work with Python 3.
I've been stuck on this function:
def send_request_json(self, request):
url = request
req = urllib.request.Request(url=url)
req.add_header('Accept', 'application/json')
try:
return json.load(urllib.request.urlopen(req))
except ValueError:
return False
When it gets to this, json responds with:
TypeError: the JSON object must be str, not 'bytes'
I read in a few places that for json.load you should pass objects (In this case an HTTPResponse object) with a .read() attached, but it doesn't work on HTTPResponse objects.
I'm at a loss as to where to go with this next, but being that my entire 1500 line script is freshly converted to Python 3, I don't feel like going back to 2.7.
Facing the same problem I solve it using decode()
...
rawreply = connection.getresponse().read()
reply = json.loads(rawreply.decode())
I recently wrote a small function to send Nexmo messages. Unless you need the full functionality of the libpynexmo code, this should do the job for you. And if you want to continue overhauling libpynexmo, just copy this code. The key is utf8 encoding.
If you want to send any other fields with your message, the full documentation for what you can include with a nexmo outbound message is here
Python 3.4 tested Nexmo outbound (JSON):
def nexmo_sendsms(api_key, api_secret, sender, receiver, body):
"""
Sends a message using Nexmo.
:param api_key: Nexmo provided api key
:param api_secret: Nexmo provided secrety key
:param sender: The number used to send the message
:param receiver: The number the message is addressed to
:param body: The message body
:return: Returns the msgid received back from Nexmo after message has been sent.
"""
msg = {
'api_key': api_key,
'api_secret': api_secret,
'from': sender,
'to': receiver,
'text': body
}
nexmo_url = 'https://rest.nexmo.com/sms/json'
data = urllib.parse.urlencode(msg)
binary_data = data.encode('utf8')
req = urllib.request.Request(nexmo_url, binary_data)
response = urllib.request.urlopen(req)
result = json.loads(response.readall().decode('utf-8'))
return result['messages'][0]['message-id']
I met the problem as well and now it pass
import json
import urllib.request as ur
import urllib.parse as par
html = ur.urlopen(url).read()
print(type(html))
data = json.loads(html.decode('utf-8'))
Since you are getting a HTTPResponse, you can use Tornado.escape and its json_decode() to convert the JSON string into a dictionary:
from tornado import escape
body = escape.json_decode(body)
From the manual:
tornado.escape.json_decode(value)
Returns Python objects for the given JSON string.
I want to return only JSON data for this view method and I'm not sure if I'm doing it the right way. Any tips would be greatly appreciated.
def helpful_click(request,object):
if request.POST and request.is_ajax():
form = HelpfulForm(request.POST)
if form.is_valid():
form.save()
return simplejson.dumps({'helpful':True})
My understanding is that every Django view should return an HttpResponse object, and you should also make sure the mime-type is set correctly:
http://jibbering.com/blog/?p=514
In a project I was working on I had something like this:
return HttpResponse(simplejson.dumps({'helpful':True}), 'application/json')
There is a JsonResponse object:
>>> from django.http import JsonResponse
>>> response = JsonResponse({'foo': 'bar'})
>>> response.content
b'{"foo": "bar"}'