I'm having trouble formatting JSON data and displaying certain fields in Python.
What I'm looking to do is only display the name and the price on a webpage via Django.
I've tried many different ways, but the only code that works right now is showing all the data, not just the name and price. The data is as follows:
{
"totalCount_str": "10134",
"items": [
{
"adjustedPrice": 306988.09,
"averagePrice": 306292.67,
"type": {
"id_str": "32772",
"href": "https://crest-tq.eveonline.com/inventory/types/32772/",
"id": 32772,
"name": "Medium Ancillary Shield Booster"
}
},
{ "..." }
],
"pageCount": 1,
"pageCount_str": "1",
"totalCount": 10134
}
item.py:
import requests
from bs4 import BeautifulSoup
# Collects the item price chart
page = requests.get('api.eveonline.com/xxxxx')
# Creates a BS4 object
soup = BeautifulSoup(page.text, 'html.parser')
item_name = soup.find(name_='')
item_price = soup.find(averagePrice='')
print(name)
print(price)
I don't know why you are trying to use an HTML parser to read JSON. BeautifulSoup has no reason to be here. Use the appropriate tool: the built in JSON library; requests will even call it for you.
data = page.json()
item = data["items"][0]
print(data["averagePrice"])
print[data["type"]["name"])
Related
I'm creating a Django rest API authentication system and I want to return data in JSON format,the main purpose of this question is : I want to add or delete users only by adding fields on json format , not in the inputs that've created :
this is my views.py :
# Rendering plant data as json :
def plant_json(request):
data=list(new_Plant.objects.values())
return JsonResponse(data,safe=False)
this is my output ( don't mind the values ) :
[{"id": 5, "name": "dqsd", "adress": "sdq", "type": "PV", "location": "dqd", "list_gateway": "gggdds", "status": "etude"}, {"id": 6, "name": "fdsfds", "adress": "fsdfds", "type": "PV", "location": "fdsfds", "list_gateway": "fdsfs", "status": "etude"}, {"id": 7, "name": "sdqdssd", "adress": "dsdsq", "type": "HYBRID", "location": "dqsdqs", "list_gateway": "dsdqss", "status": "online"}]
normally in order to add a new plant I've created a HTML template that has inputs where the user can add a plant , but what I want is that I can add a plant only by adding fields directly using json format ( using postman for example ) no need for the inputs. Thank you
What you want to use is a CreateAPIView (see docs here). All you need to do is to define the serializer you need and use it in the view.
# serializers.py
from models import Plant
from rest_framework import serializers
class PlantSerialzer(serializers.ModelSerializer):
class Meta:
model = Plant
fields = ('id', 'name', 'address', 'type', 'location', 'list_gateway', 'status')
# views.py
from my_plant_module.serializers import PlantSerializer
from rest_framework.generics import CreateAPIView
class PlantCreateView(CreateAPIView):
queryset = Plant.objects.all()
serializer_class = PlantSerializer
Using my Scala HTTP Client I retrieved a response in JSON format from an API GET call.
My end goal is to write this JSON content to an AWS S3 bucket in order to make it available as a table on RedShift running a simple AWS Glue crawler.
My thinking is to parse this JSON message and somehow converting into a Spark DataFrame, so later on I can save it to my preferred S3 location in the format of .csv, .parquet, or whatever.
The JSON file looks like this
{
"response": {
"status": "OK",
"start_element": 0,
"num_elements": 100,
"categories": [
{
"id": 1,
"name": "Airlines",
"is_sensitive": false,
"last_modified": "2010-03-19 17:48:36",
"requires_whitelist_on_external": false,
"requires_whitelist_on_managed": false,
"is_brand_eligible": true,
"requires_whitelist": false,
"whitelist": {
"geos": [],
"countries_and_brands": []
}
},
{
"id": 2,
"name": "Apparel",
"is_sensitive": false,
"last_modified": "2010-03-19 17:48:36",
"requires_whitelist_on_external": false,
"requires_whitelist_on_managed": false,
"is_brand_eligible": true,
"requires_whitelist": false,
"whitelist": {
"geos": [],
"countries_and_brands": []
}
}
],
"count": 148,
"dbg_info": {
"warnings": [],
"version": "1.18.1621",
"output_term": "categories"
}
}
}
The content I would like to map to a Dataframe is the one contained by the "categories" JSON Array.
I have managed to parse the message using json4s.JsonMethods method parse this way:
val parsedJson = parse(request) \\ "categories"
Obtaining the following:
output: org.json4s.JValue = JArray(List(JObject(List((id,JInt(1)), (name,JString(Airlines)), (is_sensitive,JBool(false)), (last_modified,JString(2010-03-19 17:48:36)), (requires_whitelist_on_external,JBool(false)), (requires_whitelist_on_managed,JBool(false)), (is_brand_eligible,JBool(true)), (requires_whitelist,JBool(false)), (whitelist,JObject(List((geos,JArray(List())), (countries_and_brands,JArray(List()))))))), JObject(List((id,JInt(2)), (name,JString(Apparel)), (is_sensitive,JBool(false)), (last_modified,JString(2010-03-19 17:48:36)), (requires_whitelist_on_external,JBool(false)), (requires_whitelist_on_managed,JBool(false)), (is_brand_eligible,JBool(true)), (requires_whitelist,JBool(false)), (whitelist,JObject(List((geos,JArray(List())), (countries_and_brands,JArray(List()))))))))
However, I am completely lost on how to proceed. I have even tried using another library for Scala called uJson:
val json = (ujson.read(request))
val tuples = json("response")("categories").arr /* <-- categories is an array */ .map { item =>
(item("id"), item("name"))
This time I have only parsed two fields for testing, but this shouldn't change much. Hence, I obtained the following structure:
tuples: scala.collection.mutable.ArrayBuffer[(ujson.Value, ujson.Value, ujson.Value, ujson.Value)] = ArrayBuffer((1,"Airlines",false,"2010-03-19 17:48:36"), (2,"Apparel",false,"2010-03-19 17:48:36"))
However, also this time I do not know how to move forward and everything I try results in errors, mostly related to format incompatibility.
Please, feel free to propose any other approach to achieve my goal even if it changes totally my workflow. I rather learn something properly. Thanks
We can use the following code to convert JSON to Spark Dataframe/Dataset
val df00 =
spark.read.option("multiline","true").json(Seq(JSON_OUTPUT).toDS())
I'm working on my first python discord bot and it's turning out to be pretty decent but I wanted to use embeds for certain responses. One of them includes sending all the features of the bots which uses many embeds. I don't have any experience about Javascript or JSON so I used a website "https://discohook.org/" to create an embed as it's GUI based. But as a result, I only get the JSON file and I couldn't find a way where I could load a JSON file and send it as an embed.
The JSON file looks something like this -
{
"content": "Bot Name",
"embeds": [
{
"description": "This command lists all the possible commands which can be used to interact with the bot.",
"author": {
"name": "Help",
"icon_url": "http://4.bp.blogspot.com/-wuUbc-OPOt4/T3ioPh2pAAI/AAAAAAAAAdM/BFFvS5fxVMY/w1200-h630-p-k-no-nu/Questionmark.jpg"
}
},
{
"description": "Bot blesses you. ",
"author": {
"name": "Bless ",
"icon_url": "https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fwallpapercave.com%2Fwp%2Fwp4775695.jpg&f=1&nofb=1"
}
},
{
"description": "Set status as idle",
"color": 16777215,
"author": {
"name": "Go Sleep",
"icon_url": "https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fmedia.forgecdn.net%2Favatars%2F170%2F652%2F636723641550179947.png&f=1&nofb=1"
}
}
]
You could probably just load the individual variables.
with open(“JSON_FILE.txt”) as json_file:
data = json.load(json_file)
data = data[“embeds”]
Then you can have some way to get the specific dictionary.
for embed in data:
pass
Then just have data defined as the dictionary you want to use.
embed = discord.Embed(description = data[“description”])
embed.set_author(name = data[“author”][“name”], icon_url = data[“author”][“icon_url”]
Maybe it've been a long time, but I have exactly the same problem and I found my own solution. Here it is:
from json import loads
from discord import Embed
# Just for our convinience let's make a function
def parse_embed_json(json_file):
embeds_json = loads(json_file)['embeds']
for embed_json in embeds_json:
embed = Embed().from_dict(embed_json)
yield embed
# And the main code looks like this
with open("bot/embeds/temp_ban_embeds.json", "r") as file:
temp_ban_embeds = parse_embed_json(file.read())
for embed in temp_ban_embeds:
await ctx.send(embed=embed)
So I found a special method from_dict. And it seems like it works with dicts we can get from json.loads method. So you can find the documentation here.
I have a following Json data to post.
{
"id": 1,
"name": "Zypher",
"price": 12.50,
"tags": [{
"tag": 1,
"tagName": "X"
},
{
"tag": 2,
"tagName": "Y"
},
{
"tag": 2,
"tagName": "Z"
}]
}
My Jmeter Test Plan is as following,
- Test Plan
- Thread Group
- Http Request Defaults
- Http Cookie Manager
- Simple Controller
- CSV Data Set Config (Sheet_1)
- Http Header Manager
- Http Request (The hard coded json was provided here as body data)
Every thing works fine. Now I want to use csv to parametrised my Json.
Sheet_1:
id,name,price
1,Zypher,12.50
I modified json with these 3 parameters and its works for me. Now I want to parametrise detail portion. I have no idea how to do this.
All I want to keep my json like this,
{
"id": ${id},
"name": ${name},
"price": ${price},
"tags": [
{
"tag": ${tag},
"tagName": ${tagName}
}]
}
How could I dynamically make json array tags for details portion from csv data? I want it to be looped as row provided in csv file.
Updated csv
id,name,price,tag,tagname
1,Zypher,12.50,7|9|11,X|Y|Z
It would be great in this format
id,name,price,tag
1,Zypher,12.50,7:X|9:Y|11:Z
tag has two properties dividing by :
You can do it using JSR223 PreProcessor and Groovy language, something like:
Given you have the following CSV file structure:
id,name,price,tag
1,Zypher,12.50,X|Y|Z
And the following CSV Data Set Config settings:
Add JSR223 PreProcessor as a child of the HTTP Request sampler and put the following code into "Script" area:
import groovy.json.JsonBuilder
def json = new JsonBuilder()
def tagsValues = vars.get("tags").split("\\|")
class Tag {int tag; String tagName }
List<Tag> tagsList = new ArrayList<>()
def counter = 1
tagsValues.each {
tagsList.add(new Tag(tag: counter, tagName: it))
counter++
}
json {
id Integer.parseInt(vars.get("id"))
name vars.get("name")
price Double.parseDouble(vars.get("price"))
tags tagsList.collect { tag ->
["tag" : tag.tag,
"tagName": tag.tagName]
}
}
sampler.addNonEncodedArgument("",json.toPrettyString(),"")
sampler.setPostBodyRaw(true)
Remove any hard-coded data from the HTTP Request sampler "Body Data" tab (it should be absolutely blank)
Run your request - JSON payload should be populated dynamically by the Groovy code:
References:
Parsing and producing JSON - Groovy
Groovy Is the New Black
Update:
for CSV format
id,name,price,tag
1,Zypher,12.50,7:X|9:Y|11:Z
Replace the below Groovy code:
List<Tag> tagsList = new ArrayList<>()
def counter = 1
tagsValues.each {
tagsList.add(new Tag(tag: counter, tagName: it))
counter++
}
with
List<Tag> tagsList = new ArrayList<>();
tagsValues.each {
String[] tag = it.split("\\:")
tagsList.add(new Tag(tag: Integer.parseInt(tag[0]), tagName: tag[1]))
}
I am building python backend for SPA (Angular) using MongoDB.
Here is what I use: Python 3.4, MongoDB 3, Flask, flask-mongoengine and flask-restful
Now I receive the following JSON from my backend:
[
{
"_id": {
"$oid": "55c737029380f82fbf52eec3"
},
"created_at": {
"$date": 1439129906376
},
"desc": "Description.....",
"title": "This is title"
},
etc...
]
And I want to receive something like that:
[
{
"_id": "55c737029380f82fbf52eec3",
"created_at": 1439129906376,
"desc": "Description.....",
"title": "This is title"
},
etc...
]
My code for now:
from flask import json
from vinnie import app
from flask_restful import Resource, Api
from vinnie.models.movie import Movie
api = Api(app)
class Movies(Resource):
def get(self):
movies = json.loads(Movie.objects().all().to_json())
return movies
api.add_resource(Movies, '/movies')
Model:
import datetime
from vinnie import db
class Movie(db.Document):
created_at = db.DateTimeField(default=datetime.datetime.now, required=True)
title = db.StringField(max_length=255, required=True)
desc = db.StringField(required=True)
def __unicode__(self):
return self.title
What is the best way to format convenient JSON for front-end?
If you are confident you want to get rid of all the similar cases, then you can certainly write code that matches that pattern. For example:
info = [
{
"_id": {
"$oid": "55c737029380f82fbf52eec3"
},
"created_at": {
"$date": 1439129906376
},
"desc": "Description.....",
"title": "This is title"
},
#etc...
]
def fix_array(info):
''' Change out dict items in the following case:
- dict value is another dict
- the sub-dictionary only has one entry
- the key in the subdictionary starts with '$'
In this specific case, one level of indirection
is removed, and the dict value is replaced with
the sub-dict value.
'''
for item in info:
for key, value in item.items():
if not isinstance(value, dict) or len(value) != 1:
continue
(subkey, subvalue), = value.items()
if not subkey.startswith('$'):
continue
item[key] = subvalue
fix_array(info)
print(info)
This will return this:
[{'title': 'This is title', 'created_at': 1439129906376, 'desc': 'Description.....', '_id': '55c737029380f82fbf52eec3'}]
Obviously, reformatting that with JSON is trivial.
I found a neat solution to my problem in flask-restful extension which I use.
It provides fields module.
Flask-RESTful provides an easy way to control what data you actually render in your response. With the fields module, you can use whatever objects (ORM models/custom classes/etc.) you want in your resource. fields also lets you format and filter the response so you don’t have to worry about exposing internal data structures.
It’s also very clear when looking at your code what data will be rendered and how it will be formatted.
Example:
from flask_restful import Resource, fields, marshal_with
resource_fields = {
'name': fields.String,
'address': fields.String,
'date_updated': fields.DateTime(dt_format='rfc822'),
}
class Todo(Resource):
#marshal_with(resource_fields, envelope='resource')
def get(self, **kwargs):
return db_get_todo() # Some function that queries the db
Flask-RESTful Output Fields Documentation