Related
I am currently developing an app using Flask + Vue.
I use flask-sqlalchemy for ORM.
When saving json to JSON type to MySQL, the action registers the data as a string versus a JSON object.
Is it possible to save the value in JSON?
Sample code:
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow
db = SQLAlchemy()
marsh = Marshmallow()
from database import db, marsh
from sqlalchemy.dialects.mysql import insert
class UserInfo(db.Model):
__tablename__ = 'user_info'
user_id = db.Column(db.String, nullable=False, primary_key=True)
json= db.Column(db.JSON, nullable=True)
def insert_on_duplicate_key_update(user_id, str_json):
json = json.dumps(str_json, ensure_ascii=False)
insert_stmt = insert(UserInfo).values({UserInfo.user_id: user_id
UserInfo.json : json })
print(json)
// print {"item1": false}
on_conflict_stmt = insert_stmt.on_duplicate_key_update(
json=insert_stmt.inserted.json)
db.engine.execute(on_conflict_stmt)
return insert_stmt
class UserInfoSchema(marsh.ModelSchema):
class Meta:
model = UserInfo
fields = ("user_id", "json")
Registration result:
select * from user_info;
{\"item1\": false}
I want to save in the following state.
{"item1": false}
I am Japanese. Sorry for poor English....
json.dumps is to make json into string
Here you make json into string:
json = json.dumps(str_json, ensure_ascii=False)
dont need. database will accept json. Possible, I use it. Maybe you need json.loads?
The answer to this question is confusing me.
Multiple Models in Django Rest Framework?
the answer is to a question of sending multipule models in a response. I have the same use case.
the author of the answer has this:
def get(self, request, format=None, **kwargs):
cart = get_cart(request)
cart_serializer = CartSerializer(cart)
another_serializer = AnotherSerializer(another_object)
return Response({
'cart': cart_serializer.data,
'another': another_serializer.data,
'yet_another_field': 'yet another value',
})
but I am keeping with the documentation.
http://www.django-rest-framework.org/api-guide/serializers/#serializing-objects
EXAMPLE FROM DOCS
serializer = CommentSerializer(comment)
serializer.data
# {'email': 'leila#example.com', 'content': 'foo bar', 'created': '2016-01-27T15:17:10.375877'}
from rest_framework.renderers import JSONRenderer
json = JSONRenderer().render(serializer.data)
json
# b'{"email":"leila#example.com","content":"foo bar","created":"2016-01-27T15:17:10.375877"}'
so which one is it? Do I JSON or not JSON. This is what I currently have.
def get(self, request, format=None):
searchcityqueryset = SearchCity.objects.all()
neighborhoodqueryset = SearchNeighborhood.objects.all()
serializedsearchcity = SearchCitySerializer(searchcityqueryset)
serializedsearchneighborhood = SearchNeighborhoodSerializer(neighborhoodqueryset)
jsonsearchcity = JSONRenderer().render(serializedsearchcity.data)
jsonsearchneighborhood = JSONRenderer().render(serializedsearchneighborhood.data)
return Response({
'searchcity': jsonsearchcity,
'searchneighborhood': jsonsearchneighborhood,
})
You don't need to do it.
From the doc:
Unlike regular HttpResponse objects, you do not instantiate Response
objects with rendered content. Instead you pass in unrendered data,
which may consist of any Python primitives.
Also JSONRenderer is default renderer class, which will be used to render.
So you can simple do this:
return Response({
'cart': cart_serializer.data,
'another': another_serializer.data,
'yet_another_field': 'yet another value',
})
I was going through django rest framework tutorial on serialization in which I got stuck at JSONRenderers and JSONParsers.Below is the code mentioned there:
from snippets.models import Snippet
from snippets.serializers import SnippetSerializer
from rest_framework.renderers import JSONRenderer
from rest_framework.parsers import JSONParser
from django.utils.six import BytesIO
snippet = Snippet(code='foo = "bar"\n')
snippet.save()
snippet = Snippet(code='print "hello, world"\n')
snippet.save()
serializer = SnippetSerializer(snippet)
serializer.data
# {'pk': 2, 'title': u'', 'code': u'print "hello, world"\n', 'linenos': False, 'language': u'python', 'style': u'friendly'}
content = JSONRenderer().render(serializer.data)
content
# '{"pk": 2, "title": "", "code": "print \\"hello, world\\"\\n", "linenos": false, "language": "python", "style": "friendly"}'
stream = BytesIO(content)
data = JSONParser().parse(stream)
I know JSONParser is used for deserializing the data and JSONRenderer is used for serializing it , but I still don't have a sound understanding of the difference between serializing and deserializing of a data. Can someone provide me a clear understanding of this.(Answer with an example is highly appreciated.)
And also how does JSONRenderer and JSONParser work?
I will start with viewsets. View and viewsets are classes in DRF, where most of application logic happens.
Eg. ModelViewSet is class responsible for CRUD operations in response to POST, PUT, PATCH, GET, DELETE HTTP methods.
Lets take look at default create method from https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/mixins.py this method create instance of your model, from data (if they are valid) send via HTTP POST method and persist them to database.
def create(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
self.perform_create(serializer)
headers = self.get_success_headers(serializer.data)
return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
def perform_create(self, serializer):
serializer.save()
This is whats happening here.
self.get_serializer() create new instance of serializer (you set serializer class before), it takes request.data as argument. request.data is (this is importent) dictionary. Dictionary is generic python data structure.
serializer.is_valid() method checks if request.data are valid. If yes you can access serializer.data - also a dictionary.
serializer.save() method creates and persist actual instance of your model (Snippet) to database.
You can directly access instance like this
instance = serializer.save()
Then you return Response object populated with serializer.data back to client.
As you can see, there is no Form data,JSON, XML, HTML etc. in viewset. You work with python data types and serializer is reponsible of "translating" dictionary to instance of your specific model and backwards.
But client send data (in your case) in HTTP request in form of JSON.
JSONParser is responsible of converting JSON to dictionary. Parsing is implemented inside request class https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/request.py , please notice that is not standard django HttpRequest model.
You can set multiple parsers, then request will choose proper one according to HTTP request header: Content-type.
Second thing is, you have to return serializer.data back to client in form of JSON, not dictionary. Thats what JSONRenderer does. It convert dict to JSON and its implemented inside Response class https://github.com/tomchristie/django-rest-framework/blob/master/rest_framework/response.py. Also you can set multiple renderers and then the proper one is choose according to accepted media type http header.
Example of full viewset definition might be:
from snippets.models import Snippet
from snippets.serializers import SnippetSerializer
from rest_framework.renderers import JSONRenderer
from rest_framework.parsers import JSONParser
class SnippetViewSet(viewsets.ModelViewSet):
queryset = Snippet.objects.all()
serializer_class = SnippetSerializer
renderer_classes = (JSONRenderer, )
parser_classes = (JSONParser,)
I am trying to write a Django Rest Framework API handler that can receive a file as well as a JSON payload. I've set the MultiPartParser as the handler parser.
However, it seems I cannot do both. If I send the payload with the file as a multi part request, the JSON payload is available in a mangled manner in the request.data (first text part until the first colon as the key, the rest is the data). I can send the parameters in standard form parameters just fine - but the rest of my API accepts JSON payloads and I wanted to be consistent. The request.body cannot be read as it raises *** RawPostDataException: You cannot access body after reading from request's data stream
For example, a file and this payload in the request body:
{"title":"Document Title", "description":"Doc Description"}
Becomes:
<QueryDict: {u'fileUpload': [<InMemoryUploadedFile: 20150504_115355.jpg (image/jpeg)>, <InMemoryUploadedFile: Front end lead.doc (application/msword)>], u'{%22title%22': [u'"Document Title", "description":"Doc Description"}']}>
Is there a way to do this? Can I eat my cake, keep it and not gain any weight?
Edit:
It was suggested that this might be a copy of Django REST Framework upload image: "The submitted data was not a file". It is not. The upload and request is done in multipart, and keep in mind the file and upload of it is fine. I can even complete the request with standard form variables. But I want to see if I can get a JSON payload in there instead.
For someone who needs to upload a file and send some data, there is no straight fwd way you can get it to work. There is an open issue in json api specs for this. One possibility i have seen is to use multipart/related as shown here, but i think its very hard to implement it in drf.
Finally what i had implemented was to send the request as formdata. You would send each file as file and all other data as text.
Now for sending the data as text you can have a single key called data and send the whole json as string in value.
Models.py
class Posts(models.Model):
id = models.UUIDField(default=uuid.uuid4, primary_key=True, editable=False)
caption = models.TextField(max_length=1000)
media = models.ImageField(blank=True, default="", upload_to="posts/")
tags = models.ManyToManyField('Tags', related_name='posts')
serializers.py -> no special changes needed, not showing my serializer here as its too lengthy because of the writable ManyToMany Field implimentation.
views.py
class PostsViewset(viewsets.ModelViewSet):
serializer_class = PostsSerializer
parser_classes = (MultipartJsonParser, parsers.JSONParser)
queryset = Posts.objects.all()
lookup_field = 'id'
You will need custom parser as shown below for parsing json.
utils.py
from django.http import QueryDict
import json
from rest_framework import parsers
class MultipartJsonParser(parsers.MultiPartParser):
def parse(self, stream, media_type=None, parser_context=None):
result = super().parse(
stream,
media_type=media_type,
parser_context=parser_context
)
data = {}
# find the data field and parse it
data = json.loads(result.data["data"])
qdict = QueryDict('', mutable=True)
qdict.update(data)
return parsers.DataAndFiles(qdict, result.files)
The request example in postman
EDIT:
see this extended answer if you want to send each data as key value pair
I know this is an old thread, but I just came across this. I had to use MultiPartParser in order to get my file and extra data to come across together. Here's what my code looks like:
# views.py
class FileUploadView(views.APIView):
parser_classes = (MultiPartParser,)
def put(self, request, filename, format=None):
file_obj = request.data['file']
ftype = request.data['ftype']
caption = request.data['caption']
# ...
# do some stuff with uploaded file
# ...
return Response(status=204)
My AngularJS code using ng-file-upload is:
file.upload = Upload.upload({
url: "/api/picture/upload/" + file.name,
data: {
file: file,
ftype: 'final',
caption: 'This is an image caption'
}
});
I send JSON and an image to create/update a product object. Below is a create APIView that works for me.
Serializer
class ProductCreateSerializer(serializers.ModelSerializer):
class Meta:
model = Product
fields = [
"id",
"product_name",
"product_description",
"product_price",
]
def create(self,validated_data):
return Product.objects.create(**validated_data)
View
from rest_framework import generics,status
from rest_framework.parsers import FormParser,MultiPartParser
class ProductCreateAPIView(generics.CreateAPIView):
queryset = Product.objects.all()
serializer_class = ProductCreateSerializer
permission_classes = [IsAdminOrIsSelf,]
parser_classes = (MultiPartParser,FormParser,)
def perform_create(self,serializer,format=None):
owner = self.request.user
if self.request.data.get('image') is not None:
product_image = self.request.data.get('image')
serializer.save(owner=owner,product_image=product_image)
else:
serializer.save(owner=owner)
Example test:
def test_product_creation_with_image(self):
url = reverse('products_create_api')
self.client.login(username='testaccount',password='testaccount')
data = {
"product_name" : "Potatoes",
"product_description" : "Amazing Potatoes",
"image" : open("local-filename.jpg","rb")
}
response = self.client.post(url,data)
self.assertEqual(response.status_code,status.HTTP_201_CREATED)
#Nithin solution works but essentially it means you are sending JSON as strings and hence not using the actual application/json inside the multipart segments.
What we want is to make the backend accept data in the below format
------WebKitFormBoundaryrga771iuUYap8BB2
Content-Disposition: form-data; name="file"; filename="1x1_noexif.jpeg"
Content-Type: image/jpeg
------WebKitFormBoundaryrga771iuUYap8BB2
Content-Disposition: form-data; name="myjson"; filename="blob"
Content-Type: application/json
{"hello":"world"}
------WebKitFormBoundaryrga771iuUYap8BB2
Content-Disposition: form-data; name="isDownscaled"; filename="blob"
Content-Type: application/json
false
------WebKitFormBoundaryrga771iuUYap8BB2--
MultiPartParser works with the above format but will treat those jsons as files. So we simply unmarshal those jsons by putting them to data.
parsers.py
from rest_framework import parsers
class MultiPartJSONParser(parsers.MultiPartParser):
def parse(self, stream, *args, **kwargs):
data = super().parse(stream, *args, **kwargs)
# Any 'File' found having application/json as type will be moved to data
mutable_data = data.data.copy()
unmarshaled_blob_names = []
json_parser = parsers.JSONParser()
for name, blob in data.files.items():
if blob.content_type == 'application/json' and name not in data.data:
mutable_data[name] = json_parser.parse(blob)
unmarshaled_blob_names.append(name)
for name in unmarshaled_blob_names:
del data.files[name]
data.data = mutable_data
return data
settings.py
REST_FRAMEWORK = {
..
'DEFAULT_PARSER_CLASSES': [
..
'myproject.parsers.MultiPartJSONParser',
],
}
This should work now.
The final bit is testing. Since the test client that ships with Django and REST doesn't support multipart JSON, we work around that by wrapping any JSON data.
import io
import json
def JsonBlob(obj):
stringified = json.dumps(obj)
blob = io.StringIO(stringified)
blob.content_type = 'application/json'
return blob
def test_simple(client, png_3x3):
response = client.post(f'http://localhost/files/', {
'file': png_3x3,
'metadata': JsonBlob({'lens': 'Sigma 35mm'}),
}, format='multipart')
assert response.status_code == 200
If you're getting an error along the lines of Incorrect type. Expected pk value, received list., with #nithin's solution, it's because Django's QueryDict is getting in the way - it's specifically structured to use a list for each entry in the dictionary, and thus:
{ "list": [1, 2] }
when parsed by MultipartJsonParser yields
{ 'list': [[1, 2]] }
which trips up your serializer.
Here is an alternative which handles this case, specifically expecting the _data key for your JSON:
from rest_framework import parsers
import json
class MultiPartJSONParser(parsers.MultiPartParser):
def parse(self, stream, *args, **kwargs):
data = super().parse(stream, *args, **kwargs)
json_data_field = data.data.get('_data')
if json_data_field is not None:
parsed = json.loads(json_data_field)
mutable_data = {}
for key, value in parsed.items():
mutable_data[key] = value
mutable_files = {}
for key, value in data.files.items():
if key != '_data':
mutable_files[key] = value
return parsers.DataAndFiles(mutable_data, mutable_files)
json_data_file = data.files.get('_data')
if json_data_file:
parsed = parsers.JSONParser().parse(json_data_file)
mutable_data = {}
for key, value in parsed.items():
mutable_data[key] = value
mutable_files = {}
for key, value in data.files.items():
mutable_files[key] = value
return parsers.DataAndFiles(mutable_data, mutable_files)
return data
It is very simple to use a multipart post and a regular view, if this is an option.
You send the json as a field and files as files, then process in one view.
Here is a simple python client and a Django server:
Client - sending multiple files and an arbitrary json-encoded object:
import json
import requests
payload = {
"field1": 1,
"manifest": "special cakes",
"nested": {"arbitrary":1, "object":[1,2,3]},
"hello": "word" }
filenames = ["file1","file2"]
request_files = {}
url="example.com/upload"
for filename in filenames:
request_files[filename] = open(filename, 'rb')
r = requests.post(url, data={'json':json.dumps(payload)}, files=request_files)
Server - consuming the json and saving the files:
#csrf_exempt
def upload(request):
if request.method == 'POST':
data = json.loads(request.POST['json'])
try:
manifest = data['manifest']
#process the json data
except KeyError:
HttpResponseServerError("Malformed data!")
dir = os.path.join(settings.MEDIA_ROOT, "uploads")
os.makedirs(dir, exist_ok=True)
for file in request.FILES:
path = os.path.join(dir,file)
if not os.path.exists(path):
save_uploaded_file(path, request.FILES[file])
else:
return HttpResponseNotFound()
return HttpResponse("Got json data")
def save_uploaded_file(path,f):
with open(path, 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
I'd just like to add to #Pithikos's answer by modifying the parser to accept lists as well, in line with how DRF parses lists in serializers in utils/html#parse_html_list
class MultiPartJSONParser(parsers.MultiPartParser):
def parse(self, stream, *args, **kwargs):
data = super().parse(stream, *args, **kwargs)
# Any 'File' found having application/json as type will be moved to data
mutable_data = data.data.copy()
unmarshaled_blob_names = []
json_parser = parsers.JSONParser()
for name, blob in data.files.items():
if blob.content_type == 'application/json' and name not in data.data:
parsed = json_parser.parse(blob)
if isinstance(parsed, list):
# need to break it out into [0], [1] etc
for idx, item in enumerate(parsed):
mutable_data[name+f"[{str(idx)}]"] = item
else:
mutable_data[name] = parsed
unmarshaled_blob_names.append(name)
for name in unmarshaled_blob_names:
del data.files[name]
data.data = mutable_data
return data
The following code worked for me.
from django.core.files.uploadedfile import SimpleUploadedFile
import requests
from typing import Dict
with open(file_path, 'rb') as f:
file = SimpleUploadedFile('Your-Name', f.read())
data: Dict[str,str]
files: Dict[str,SimpleUploadedFile] = {'model_field_name': file}
requests.put(url, headers=headers, data=data, files=files)
requests.post(url, headers=headers, data=data, files=files)
'model_field_name' is the name of the FileField or ImageField in your Django model. You can pass other data as name or location as usual by using data parameter.
Hope this helps.
This work for me:
class FileUpload(APIView):
parser_classes = [MultiPartParser]
authentication_classes = [JWTAuthentication]
def post(self, request, filename, format=None):
file = request.data['file']
data = json.loads(request.POST['form'])
#.... just do.....
.
.
.
frontend part: example with fetch (vue frontend)
let data = await new FormData(); // creates a new FormData object
data.append("file", this.files); // add your file to form data
data.append('form',JSON.stringify(body)) //add your json
fetch(`https://endpoint/FileUpload/${body.nombre}`, {
method: 'POST',
body: data,
headers: {Authorization: `Bearer ${accessToken}`}
})
I hope this helps.
I want to serialize my queryset, and I want it in a format as this view outputs:
class JSONListView(ListView):
queryset = Users.objects.all()
def get(self, request, *args, **kwargs):
return HttpResponse(json.dumps({'data': [['bar','foo','bar','foo'],['foo','bar','foo','bar']]}, indent=4), content_type='application/json')
I simply don't know how to output the queryset instead of the manual data in the example.
I've tried
json.dumps({"data": self.get_queryset()})
and
serializers.serialize("json", {'data': self.get_queryset()})
but it won't work. What am I doing wrong? Do I need to make a custom JSON Encoder?
You can use JsonResponse with values. Simple example:
from django.http import JsonResponse
def some_view(request):
data = list(SomeModel.objects.values()) # wrap in list(), because QuerySet is not JSON serializable
return JsonResponse(data, safe=False) # or JsonResponse({'data': data})
Or another approach with Django's built-in serializers:
from django.core import serializers
from django.http import HttpResponse
def some_view(request):
qs = SomeModel.objects.all()
qs_json = serializers.serialize('json', qs)
return HttpResponse(qs_json, content_type='application/json')
In this case result is slightly different (without indent by default):
[
{
"model": "some_app.some_model",
"pk": 1,
"fields": {
"name": "Elon",
"age": 48,
...
}
},
...
]
I have to say, it is good practice to use something like marshmallow to serialize queryset.
...and a few notes for better performance:
use pagination if your queryset is big;
use objects.values() to specify list of required fields to avoid serialization and sending to client unnecessary model's fields (you also can pass fields to serializers.serialize);
It didn't work, because QuerySets are not JSON serializable.
1) In case of json.dumps you have to explicitely convert your QuerySet to JSON serializable objects:
class Model(model.Model):
def as_dict(self):
return {
"id": self.id,
# other stuff
}
And the serialization:
dictionaries = [ obj.as_dict() for obj in self.get_queryset() ]
return HttpResponse(json.dumps({"data": dictionaries}), content_type='application/json')
2) In case of serializers. Serializers accept either JSON serializable object or QuerySet, but a dictionary containing a QuerySet is neither. Try this:
serializers.serialize("json", self.get_queryset())
Read more about it here:
https://docs.djangoproject.com/en/dev/topics/serialization/
For a efficient solution, you can use .values() function to get a list of dict objects and then dump it to json response by using i.e. JsonResponse (remember to set safe=False).
Once you have your desired queryset object, transform it to JSON response like this:
...
data = list(queryset.values())
return JsonResponse(data, safe=False)
You can specify field names in .values() function in order to return only wanted fields (the example above will return all model fields in json objects).
To return the queryset you retrieved with queryset = Users.objects.all(), you first need to serialize them.
Serialization is the process of converting one data structure to another. Using Class-Based Views, you could return JSON like this.
from django.core.serializers import serialize
from django.http import JsonResponse
from django.views.generic import View
class JSONListView(View):
def get(self, request, *args, **kwargs):
qs = User.objects.all()
data = serialize("json", qs)
return JsonResponse(data)
This will output a list of JSON. For more detail on how this works, check out my blog article How to return a JSON Response with Django. It goes into more detail on how you would go about this.
If the goal is to build an API that allow you to access your models in JSON format I recommend you to use the django-restframework that is an enormously popular package within the Django community to achieve this type of tasks.
Django Rest Framework Website
Github
It include useful features such as Pagination, Defining Serializers, Nested models/relations and more. Even if you only want to do minor Javascript tasks and Ajax calls I would still suggest you to build a proper API using the Django Rest Framework instead of manually defining the JSON response.
Another way to turn queryset into JSON, is appending necessary elements to an empty list with loop. It provides to design customizable JSON.
queryset = Users.objects.all()
output = []
for query in queryset:
output.append('id': query.id, 'name': query.name, etc...)
return JSONResponse(output, safe=False)
Try this:
class JSONListView(ListView):
queryset = Users.objects.all()
def get(self, request, *args, **kwargs):
data = {}
data["users"] = get_json_list(queryset)
return JSONResponse(data)
def get_json_list(query_set):
list_objects = []
for obj in query_set:
dict_obj = {}
for field in obj._meta.get_fields():
try:
if field.many_to_many:
dict_obj[field.name] = get_json_list(getattr(obj, field.name).all())
continue
dict_obj[field.name] = getattr(obj, field.name)
except AttributeError:
continue
list_objects.append(dict_obj)
return list_objects
from django.http import JsonResponse
def SomeFunction():
dict1 = {}
obj = list( Mymodel.objects.values() )
dict1['data']=obj
return JsonResponse(dict1)
Try this code for Django