Data sample:
import pandas as pd
patients_df = pd.read_json('C:/MyWorks/Python/Anal/data_sample.json', orient="records", lines=True)
patients_df.head()
//in python
//my json data sample
"data1": {
"id": "myid",
"seatbid": [
{
"bid": [
{
"id": "myid",
"impid": "1",
"price": 0.46328014,
"adm": "adminfo",
"adomain": [
"domain.com"
],
"iurl": "url.com",
"cid": "111",
"crid": "1111",
"cat": [
"CAT-0101"
],
"w": 00,
"h": 00
}
],
"seat": "27"
}
],
"cur": "USD"
},
What I want to do is to check if there is a "cat" value in my very large JSON data.
The "cat" value may/may not exist, but I'm trying to use Python Pandas to check it.
for seatbid in patients_df["win_res"]:
for bid in seatbid["seatbid"]:
I tried to access JSON data while writing a loop like that, but it's not being accessed properly.
I simply want to check if "cat" exist or not.
You can use python's json library as follows:
import json
patient_data = json.loads(patientJson)
if "cat" in student:
print("Key exist in JSON data")
else
print("Key doesn't exist in JSON data")
Related
I am trying to get all IPs from a JSON file using Python 2.7.5
However I can not manage to do it correctly.
Do someone have an advice how I can receive all IPs from ('addressPrefixes') in a txt file?
Here is the code I already got to download the json file:
import urllib
import json
from urllib import urlopen
testfile = urllib.URLopener()
testfile.retrieve("https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-
DA13A5DE5B63/ServiceTags_Public_20210426.json", "AzureIPs.json")
print("---SUCCESSFULLY RECEIVED MICROSOFT AZURE IPS---")
with open('AzureIPs.json','r') as f:
data = json.load(f)
the JSON file contains many IPs and IP Ranges and looks like this:
{
"changeNumber": 145,
"cloud": "Public",
"values": [
{
"name": "ActionGroup",
"id": "ActionGroup",
"properties": {
"changeNumber": 9,
"region": "",
"regionId": 0,
"platform": "Azure",
"systemService": "ActionGroup",
"addressPrefixes": [
"13.66.60.119/32",
"13.66.143.220/30",
"13.66.202.14/32",
"13.66.248.225/32",
"13.66.249.211/32",
"13.67.10.124/30",
"13.69.109.132/30",
"13.71.199.112/30",
"13.77.53.216/30",
"13.77.172.102/32",
"13.77.183.209/32",
"13.78.109.156/30",
"13.84.49.247/32",
"2603:1030:c06:400::978/125",
"2603:1030:f05:402::178/125",
"2603:1030:1005:402::178/125",
"2603:1040:5:402::178/125",
"2603:1040:207:402::178/125",
"2603:1040:407:402::178/125",
"2603:1040:606:402::178/125",
"2603:1040:806:402::178/125",
"2603:1040:904:402::178/125",
"2603:1040:a06:402::178/125",
"2603:1040:b04:402::178/125",
"2603:1040:c06:402::178/125",
"2603:1040:d04:800::f8/125",
"2603:1040:f05:402::178/125",
"2603:1040:1104:400::178/125",
"2603:1050:6:402::178/125",
"2603:1050:403:400::1f8/125"
],
"networkFeatures": [
"API",
"NSG",
"UDR",
"FW"
]
}
},
{
"name": "ApplicationInsightsAvailability",
"id": "ApplicationInsightsAvailability",
"properties": {
"changeNumber": 2,
"region": "",
"regionId": 0,
"platform": "Azure",
"systemService": "ApplicationInsightsAvailability",
"addressPrefixes": [
"13.86.97.224/27",
"13.86.98.0/27",
"13.86.98.48/28",
"13.86.98.64/28",
"20.37.156.64/27",
"20.37.192.80/29",
"20.38.80.80/28",
"20.40.104.96/27",
"20.40.104.128/27",
"20.40.124.176/28",
"20.40.124.240/28",
"20.40.125.80/28",
"20.40.129.32/27",
"20.40.129.64/26",
"20.40.129.128/27",
"20.42.4.64/27",
"20.42.35.32/28",
"20.42.35.64/26",
"20.42.35.128/28",
"20.42.129.32/27",
"20.43.40.80/28",
"20.43.64.80/29",
"20.43.128.96/29",
"20.45.5.160/27",
"20.45.5.192/26",
"20.189.106.64/29",
"23.100.224.16/28",
"23.100.224.32/27",
"23.100.224.64/26"
],
"networkFeatures": [
"API",
"NSG",
"UDR",
"FW"
]
}
},
{
"name": "AzureActiveDirectory",
"id": "AzureActiveDirectory",
"properties": {
"changeNumber": 8,
"region": "",
"regionId": 0,
"platform": "Azure",
"systemService": "AzureAD",
"addressPrefixes": [
"13.64.151.161/32",
"13.66.141.64/27",
"13.67.9.224/27",
"13.69.66.160/27",
"13.69.229.96/27",
"13.70.73.32/27"
],
"networkFeatures": [
"API",
"NSG",
"UDR",
"FW",
"VSE"
]
}
}
Thank you for your time.
import urllib
import json
from urllib import urlopen
testfile = urllib.URLopener()
testfile.retrieve("https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-
DA13A5DE5B63/ServiceTags_Public_20210426.json", "AzureIPs.json")
print("---SUCCESSFULLY RECEIVED MICROSOFT AZURE IPS---")
with open('AzureIPs.json','r') as f:
data = json.load(f)
################# CHANGES AFTER THIS LINE #################
ips = []
values = data['values']
for block in values:
ips.append(block.properties.addressPrefixes)
However you will get 2D array using this approach, if you need 1D array and not separate block of IPs from each corresponding block in values, you can use following code to flatten the array.
import numpy as np
2DArray = np.array(ips)
1DArray = 2DArray.flatten()
I have about 100 JSON files, all titled with different dates and I need to merge them into one CSV file that has headers "date", "real_name", "text".
There are no dates listed in the JSON itself, and the real_name is nested. I haven't worked with JSON in a while and am a little lost.
The basic structure of the JSON looks more or less like this:
Filename: 2021-01-18.json
[
{
"client_msg_id": "xxxx",
"type": "message",
"text": "THIS IS THE TEXT I WANT TO PULL",
"user": "XXX",
"user_profile": {
"first_name": "XXX",
"real_name": "THIS IS THE NAME I WANT TO PULL",
"display_name": "XXX",
"is_restricted": false,
"is_ultra_restricted": false
},
"blocks": [
{
"type": "rich_text",
"block_id": "yf=A9",
}
]
}
]
So far I have
import glob
read_files = glob.glob("*.json")
output_list = []
all_items = []
for f in read_files:
with open(f, "rb") as infile:
output_list.append(json.load(infile))
data = {}
for obj in output_list[]
data['date'] = f
data['text'] = 'text'
data['real_name'] = 'real_name'
all_items.append(data)
Once you've read the JSON object, just index into the dictionaries for the data. You might need obj[0]['text'], etc., if your JSON data is really in a list in each file, but that seems odd and I'm assuming your data was pasted from output_list after you'd collected the data. So assuming your file content is exactly like below:
{
"client_msg_id": "xxxx",
"type": "message",
"text": "THIS IS THE TEXT I WANT TO PULL",
"user": "XXX",
"user_profile": {
"first_name": "XXX",
"real_name": "THIS IS THE NAME I WANT TO PULL",
"display_name": "XXX",
"is_restricted": false,
"is_ultra_restricted": false
},
"blocks": [
{
"type": "rich_text",
"block_id": "yf=A9",
}
]
}
test.py:
import json
import glob
from pathlib import Path
read_files = glob.glob("*.json")
output_list = []
all_items = []
for f in read_files:
with open(f, "rb") as infile:
output_list.append(json.load(infile))
data = {}
for obj in output_list:
data['date'] = Path(f).stem
data['text'] = obj['text']
data['real_name'] = obj['user_profile']['real_name']
all_items.append(data)
print(all_items)
Output:
[{'date': '2021-01-18', 'text': 'THIS IS THE TEXT I WANT TO PULL', 'real_name': 'THIS IS THE NAME I WANT TO PULL'}]
I convert excel file to JSON , to import it into my firebase DB.
On conversion, I have the JSON data in below format
[
{
"ProductNumber": "7381581",
"SKU": "test3",
},
{
"ProductNumber": "7381582",
"SKU": "test",
},
{..}
]
But I need it like this
{
"7381581" :{
"ProductNumber": "7381581",
"SKU": "test3",
},
"7381582":{
"ProductNumber": "7381582",
"SKU": "test",
},{..}
}
How can I make changes to the spreadsheet records to get the JSON in the above format ? (OR)
How should I add the key values to JSON dynamically?
You can use reduce as suggested to iterate over the original array and transform it into an object.
data.reduce((prev, current) => {
prev[current.ProductNumber] = current;
return prev;
}, {});
You can see a working example in the playground here.
Lets say we have our MongoDB and we want to backup the data into a .json file example for an output file : database.json and inside :
{
"collections": [
{"name": "admin"},
{"name": "class"},
{"name": "lesson"},
{"name": "message"},
{"name": "room"},
{"name": "student"},
{"name": "subject"},
{"name": "teacher"}
],
"subjects": [
{
"name": "Null",
"color": "#FFFFFF"
},
{
"name": "Design Art",
"color": "#82B9D6"
},
{
"name": "Plastic Art",
"color": "#a3db05"
},
{
"name": "Media And Production",
"color": "#522a64"
}, //...there is a continue to this file ....
}
each collection should be added to the collections and for each collection there should be an array of all the info inside it (like above)
I'm using python 3.4 with the pymongo driver.
What is the best way to get all the info from the DB , create the JSON object and insert it to a new .json file
I found this way of doing it :
import json
from pymongo import MongoClient
from pprint import pprint
def main():
with open('../config/database.json') as database_config:
config = json.load(database_config)
client = MongoClient(config["mongodb"])
db = client[config["database"]]
data = dict()
data["collections"] = db.collection_names()
for collection_name in db.collection_names():
data[collection_name] = get_collection(db, collection_name)
pprint(data)
insert_data_to_file(data)
def get_collection(db, collection_name):
collection_list = []
collection = db[collection_name]
cursor = collection.find({})
for document in cursor:
_id = document.pop('_id')
document['_id'] = str(_id)
collection_list.append(document)
return collection_list
def insert_data_to_file(data):
with open('database.json', 'x') as database:
json.dump(data, database, sort_keys=True)
main()
I'm using the next Json
{
"ID": 8,
"MenuItems": [
{
"ID": 38,
"Name": "Home",
"URL": "{\"PageLayout\":\"Home\",\"Icon\":\"home\"}",
"Culture": "en",
"Children": null
},
{
"ID": 534,
"Name": "GUIDE ",
"URL": "{''PageLayout'':''Page A'', ''Icon'':''A''}",
"MenuType": 1,
"PageID": 0,
"Culture": "en",
"Children": [
{
"ID": 6,
"Name": "Form A",
"URL": "[''Type'':''Form A'',''Icon'':''Form'',''ItemID'':\"358\"]",
"Culture": "he",
"RuleID": 0
},
{
"ID": 60,
"Name": "Drama",
"URL": "[''Type'':''Form B'',''Icon'':''Form'',''ItemID'':\"3759\"]",
"Culture": "en",
"RuleID": 0
}
]
}
]
}
i'm using Groovy script in soapUI and i need to:
Assert the exitance of node that has the name GUIDE
Extract a list of all Itemsid
You can parse the JSON content using JsonSlurper and then work with the results like so:
import groovy.json.JsonSlurper
// Assuming your JSON is stored in "jsonString"
def jsonContent = new JsonSlurper().parseText(jsonString)
// Assert node exists with name GUIDE
assert(jsonContent.MenuItems.Name.contains("GUIDE"))
// Get list of ItemIDs
def itemList = jsonContent.MenuItems.Children.URL.ItemID[0].toList()
// List the items
itemList.each {log.info it}
Note that the above will fail given your current example, because of a few issues. Firstly, Name contains "GUIDE " (trailing space) rather than "GUIDE" (so you will need to adapt accordingly). Secondly, it is invalid JSON; the URL nodes have various erroneous characters.
On a side note, if you first need to retrieve your JSON from the response associated with a previous TestStep (say one called "SendMessage") within the existing testCase, this can be done as follows:
def jsonString = context.testCase.getTestStepByName("SendMessage").testRequest.response.getContentAsString()