I'm somewhat new to Django and am trying to get Content Security Policy reporting running on my application and am running into some issues with parsing and storing the JSON violation output. I have all reports POSTing to /csp_reports/ and have built a new app called security_report and added the following to my urls.py:
url(r'^csp_report/$', 'security_report.views.secreport'),
My models look like this:
import os
from django.db import models
from django.contrib.auth.models import User
from django.conf import settings
class security_report(models.Model):
id=models.AutoField(primary_key=True)
received=models.DateTimeField(null=True)
csp_report=models.CharField()
blocked_uri=models.CharField()
column_number=models.CharField()
document_uri=models.CharField()
line_number = models.IntegerField()
original_policy= models.CharField()
referrer = models.CharField()
status_code = models.IntegerField()
violated_directive = models.CharField()
source_file = models.CharField()
script_sample = models.CharField()
class Meta:
app_label = "events"
And my views.py which is not complete and not working is below. I am stuck on how to actually grab each individual parameter and dump it into my database. Is json.loads the right way to do this? If so, how do I perform the save()? I am only testing the blocked_uri in the view for now to test it out. I just keep hitting my except: and getting "not saved"
from security_report.models import security_report
from django.utils import simplejson
from django.http import HttpResponse, HttpRequest
from django.views.decorators.csrf import csrf_exempt, csrf_protect
import json
#csrf_exempt
def secreport(request):
if request.method == "POST":
json_data = simplejson.loads(request.raw_post_data)
try:
data = json_data['csp-report']
blocked_uri = data['blocked-uri']
document_uri = data['document-uri']
referrer = data['referrer']
script_sample = data['script-sample']
violated_directive = data['violated-directive']
source_file = data['source-file']
b = security_report()
b.blocked_uri = blocked_uri
b.save()
response = ("saved to database!")
except:
response = "not saved"
return HttpResponse(response)
else:
return HttpResponse("not a post request")
Related
I would like to create a django rest api that receives a csv file and then response the file in a json format. How do I achieve this without using models and anything related with databases? I don't want to use a database because it's assumed that the csv files will have different structure every time
This is my first django project and I've seen lots of tutorials, even I did the first tutorial from django website but I can't understand how to do this task without the database.
Thanks!
since you have not tried anything on your own Here is how you can do it
views.py
from rest_franework.generics import CreateAPIView
class ReadCSVView(CreateAPIView):
# permission_classes = [IsAuthenticated]
serializer_class = ReadCSVSerializer
queryset = ''
def perform_create(self, serializer):
file = serializer.validated_data['file']
decoded_file = file.read().decode()
io_string = io.StringIO(decoded_file)
reader = csv.reader(io_string)
next(reader) # incase you want to skip first row else remove this
return reader
def create(self, request, *args, **kwargs):
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
final_data = []
for row in self.perform_create(serializer):
if row[0]=="jack":
#do your logic here
final_data.append(row)
return Response(final_data, status=status.HTTP_201_CREATED)
just create one serializers to read csv.
serializers.py
from rest_framework import serializers
class ReadCSVSerializer(serializers.Serializer):
file = serializers.FileField()
now go to your urls.py and call the view class this way
urlpatterns = [
path("read-csv",views.ReadCSVView.as_view(),name="csv")]
hope this clarifies your doubt
I am new to django rest framework (DRF) and I need to POST some data using function based views (FDV). I successfully used GET method using this way but have no idea how to use POST method to add values to database.
# app/models.py
class Item(models.Model):
name = models.CharField(max_length=50)
quantity = models.IntegerField()
price = models.FloatField()
# app/urls.py
urlpatterns = [
path('', views.get_data_list, name='list'),
path('post_val/', views.post_data, name='post_val'), # need to implement
]
# app/serializers.py
class ItemSerializer(serializers.ModelSerializer):
class Meta:
model = Item
fields = ('id', 'name', 'quantity','price')
# app/views.py
from django.http.response import JsonResponse
from rest_framework.parsers import JSONParser
from .models import Item
from .serializers import ItemSerializer
from rest_framework.decorators import api_view
#api_view(['GET',])
def get_data_list(request):
if request.method == 'GET':
items = Item.objects.all()
items_serializer = ItemSerializer(items, many=True)
return JsonResponse(items_serializer.data, safe=False)
#api_view(['POST',])
def post_data(request):
#TO DO
If I want to add this new data like this one {name:"Television", quantity:15, price:999.99} to Item table using POST method, How do we do it in FDV?
Try this
#api_view(['POST',])
def post_data(request):
serializer = ItemSerializer(data=request.data)
serializer.is_valid(raise_exception=True)
serializer.save()
return Response(serializer.data)
On a side note, you can use DRF's Response object for returning the response. It will automatically convert to JSON or XML based on your config.
Import it via
from rest_framework.response import Response
django-rest-framework puts the posted data in request.data
So in short you can retrieve your data from there and create your object:
#api_view(['POST',])
def post_data(request):
item_serializer = ItemSerializer(data=request.data)
if item_serializer.is_valid():
item = item_serializer.save()
I am quite new to django. I am trying to convert sql data fetched from a remote postgresql database into JSON so as to use it in react. But while dumping the data it throws as error.
`AttributeError: 'str' object has no attribute 'get'`
I tried many versions of dumping sql data into json like coneverting data into list and using RealDictCursor but each one of them throws a new error.
Views.py
from django.shortcuts import render, get_object_or_404
from django.http import JsonResponse
from django.http import HttpResponse
from .marketer import marketer
def marketer_list(request):
return JsonResponse(marketer)
marketer.py (function to fetch data and establish the connection)
from django.shortcuts import render, get_object_or_404
import json
import psycopg2
from psycopg2.extras import RealDictCursor
def marketer(self):
connection = psycopg2.connect(user = "db-user",
password = "*****",
host = "18.23.42.2",
port = "5432",
database = "db-name")
cursor = connection.cursor(cursor_factory = RealDictCursor)
postgreSQL_select_Query = "select id from auth_permission"
result = cursor.execute(postgreSQL_select_Query)
#print("Selecting rows from mobile table using cursor.fetchall")
#mobile = dictfetchall(result)
#items = [dict(zip([key[0] for key in cursor.description], row)) for
row in result]
return json.dumps(cursor.fetchall(), indent=2)
Error at Url page
AttributeError: 'str' object has no attribute 'get'
or
in some other methods
is not JSON serializable
I am trying to save data in form of JSON (returned as result from POST request)
def get_data(...):
...
try:
_r = requests.post(
_url_list,
headers=_headers
)
return _r.json()
except Exception as ee:
print('Could not get data: {}'.format(ee))
return None
Into a table in SQLITE database as backend.
def add_to_flight_data(_data):
if _data:
try:
new_record = FlightData(data=_data)
db.session.add(new_record)
db.session.commit()
print('Data instertedto DB!')
return "Success"
except Exception as e:
print('Data NOT instertedto DB! {}'.format(e))
pass
This is my simple flask code
import os
import time
import auth
import json
import requests
import datetime
from flask import Flask
from flask_marshmallow import Marshmallow
from flask_sqlalchemy import SQLAlchemy
# from safrs.safrs_types import JSONType
project_dir = os.path.dirname(os.path.abspath(__file__))
database_file = "sqlite:///{}".format(os.path.join(project_dir, "2w.sqlite"))
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = database_file
db = SQLAlchemy(app)
ma = Marshmallow(app)
class FlightData(db.Model):
id = db.Column(db.Integer, primary_key=True)
created = db.Column(db.DateTime, server_default=db.func.now())
json_data = db.Column(db.JSONType, default={})
def __init__(self, data):
self.data = data
It seems like there is perhaps no option to save JSON in sqlite
json_data = db.Column(db.JSONType, default={})
Please ADVISE
Thanks.
I believe that you should be using db.JSON, not db.JSONType as there is no such column type in sqlalchemy.
Regardless of that, SQLite has no JSON data type, so sqlalchemy won't be able to map columns of type db.JSON onto anything. According to the documentation only Postgres and some MySQL are supported. There is support for JSON in SQLite with the JSON1 extension, but sqlalchemy will not be able to make use of it.
Your best bet then is to declare the column as db.Text and use json.dumps() to jsonify the data on write. Alternatively modify your get_data() function to check for a JSON response (check the Content-type header or try calling _r.json() and catching exceptions), and then return _r.content which will already be a JSON string.
Use json.loads() to read data back from the db.
Here I want to store the data from the list given on a website page. If I'm running the commands
response.css('title::text').extract_first() and
response.css("article div#section-2 li::text").extract()
individually in the scrapy shell it is showing expected output in shell.
Below is my code which is not storing data in json or csv format:
import scrapy
class QuotesSpider(scrapy.Spider):
name = "medical"
start_urls = ['https://medlineplus.gov/ency/article/000178.html/']
def parse(self, response):
yield
{
'topic': response.css('title::text').extract_first(),
'symptoms': response.css("article div#section-2 li::text").extract()
}
I have tried to run this code using
scrapy crawl medical -o medical.json
You need to fix your URL, it is https://medlineplus.gov/ency/article/000178.htm and not https://medlineplus.gov/ency/article/000178.html/.
Also, and more importantly, you need to define an Item class and yield/return it from the parse() callback of your spider:
import scrapy
class MyItem(scrapy.Item):
topic = scrapy.Field()
symptoms = scrapy.Field()
class QuotesSpider(scrapy.Spider):
name = "medical"
allowed_domains = ['medlineplus.gov']
start_urls = ['https://medlineplus.gov/ency/article/000178.htm']
def parse(self, response):
item = MyItem()
item["topic"] = response.css('title::text').extract_first()
item["symptoms"] = response.css("article div#section-2 li::text").extract()
yield item