Json file parsing string indices must be integers, not 'str' error - json

I'm having an issue while coding a small bot in python. I have a Json file that I retrieved from an api looking like the following:
> {
> "10000NFTUSDT": {
> "ret_code": 0,
> "ret_msg": "OK",
> "ext_code": "",
> "ext_info": "",
> "result": \[
> {
> "symbol": "10000NFTUSDT",
> "period": "60",
> "start_at": 1671559200,
> "open": 0.004533,
> "high": 0.004538,
> "low": 0.004523,
> "close": 0.004535
> },
> {
> "symbol": "10000NFTUSDT",
> "period": "60",
> "start_at": 1671562800,
> "open": 0.004535,
> "high": 0.004537,
> "low": 0.004531,
> "close": 0.004533
> },
> {
> "symbol": "10000NFTUSDT",
> "period": "60",
> "start_at": 1671566400,
> "open": 0.004533,
> "high": 0.00454,
> "low": 0.004533,
> "close": 0.004534
> },
I'm trying to extract the close price from each coins (to then compare it them with each other) but I'm stuck at the extracting process. My main file look like this:
import json
from func_cointegration import get_cointegrated_pairs
\#STEP 3 Find Cointegrated pairs
print("Calculating co-integration...")
with open("1_price_list.json") as json_file:
price_data = json.load(json_file)
if len(price_data) > 0:
coint_pairs = get_cointegrated_pairs(price_data)
I'm calling get_cointegrated_pairs that use extract_close_prices
import math
#Put close prices into a list
def extract_close_prices(prices):
#trying to extract close price
close_prices = []
for price_values in prices:
print(type(price_values))
if math.isnan(price_values["close"]):
return []
close_prices.append(price_values["close"])
print(close_prices)
return close_prices
#Calculate cointegrated pairs
def get_cointegrated_pairs(prices):
#Loop through all the coin and check co-integration.
coint_pair_list = []
included_list = []
for sym_1 in prices.keys():
#check each coin against the first (sym1)
for sym_2 in prices.keys():
if sym_2 != sym_1:
#Get unique combination id and ensure one off check
sorted_characters = sorted(sym_1 + sym_2)
unique = "".join(sorted_characters)
if unique in included_list:
break
#get close prices
series_1 = extract_close_prices(prices[sym_1])
series_2 = extract_close_prices(prices[sym_2])
I don't understand why I can't access "price_values" with the key "close". I though that opening the json file with load would ensure it loads as a dictionary, I'm following a tutorial and the guy run a similar code without this "typeError" issue. I want to feed my "close_price" list with all the close price from my json file. Any ideas on what i'm doing wrong?
Traceback (most recent call last):
File "/Users/xxxxxxxx/PycharmProjects/StatArbBot/Strategy/main.py", line 27, in <module>
coint_pairs = get_cointegrated_pairs(price_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/xxxxx/PycharmProjects/StatArbBot/Strategy/func_cointegration.py", line 35, in get_cointegrated_pairs
series_1 = extract_close_prices(prices[sym_1])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/xxxxx/PycharmProjects/StatArbBot/Strategy/func_cointegration.py", line 12, in extract_close_prices
if math.isnan(price_values["close"]):
~~~~~~~~~~~~^^^^^^^^^
TypeError: string indices must be integers, not 'str'

So I tried to replicate your code and issue.
Turns out that you were passing a dictionary and trying to iterate through it.
for price_values in prices: in your extract_close_prices function.
So instead you need to iterate to the list nested in the dictionary under the key result like so:
for price_values in prices['result']:
Then you try to see if the list item is not a number and since it's an object (dictionary technically) with if math.isnan(price_values): then you get an empty list returned.
instead you need to check if the value of the key close is a number like so: if math.isnan(price_values["close"])
After that the rest of your code should work.
Here is a working copy I created from your posted question.
https://gist.github.com/HH0718/a6dcfa07c0eb2f83070f01544d04270c
Original Answer below:
You are attempting to get an item from a list that's nested in a JSON file.
Therefore, you need to provide the index of the item you are looking for. Once you provide the index of that item within the list you can get the value of that object.
Notice how "result" is a list:
"result": [ { "symbol": "10000NFTUSDT", "period": "60", "start_at": 1671559200, "open": 0.004533, "high": 0.004538, "low": 0.004523, "close": 0.004535 },...
as indicated by the [
for example:
response.get("10000NFTUSDT").get("result")[0].get["close"]
will get you,0.004535

Related

Change name of main row Rails in JSON

So i have a json:
{
"code": "Q0934X",
"name": "PIDBA",
"longlat": "POINT(23.0 33.0)",
"altitude": 33
}
And i want to change the column code to Identifier
The wished output is this
{
"Identifier": "Q0934X",
"name": "PIDBA",
"longlat": "POINT(23.0 33.0)",
"altitude": 33
}
How can i do in the shortest way? Thanks
It appears that both "the json" you have and your desired result are JSON strings. If the one you have is json_str you can write:
json = JSON.parse(json_str).tap { |h| h["Identifier"] = h.delete("code") }.to_json
puts json
#=> {"name":"PIDBA","longlat":"POINT(23.0 33.0)","altitude":33,"Identifier":"Q0934X"}
Note that Hash#delete returns the value of the key being removed.
Perhaps transform_keys is an option.
The following seems to work for me (ruby 2.6):
json = JSON.parse(json_str).transform_keys { |k| k === 'code' ? 'Identifier' : k }.to_json
But this may work for Ruby 3.0 onwards (if I've understood the docs):
json = JSON.parse(json_str).transform_keys({ 'code': 'Identifier' }).to_json

JMeter: How can I randomize post body data several times?

I have a post body data as:
"My data": [{
"Data": {
"var1": 6.66,
"var2": 8.88
},
"var3": 9
}],
Here, if I post these details on POST DATA body, it will call "My Data" just once. I want to make it random as starting from 1 to 10 times so that "My data" is running for several times but randomly. If the random value is 2, then "My data" should run twice.
Help appreciated!
If you need to generate more blocks like this one:
{
"Data": {
"var1": 6.66,
"var2": 8.88
},
"var3": 9
}
It can be done using JSR223 PreProcessor and the following code:
def myData = []
1.upto(2, {
def entry = [:]
entry.put('Data', [var1: 6.66, var2: 8.88])
entry.put('var3', '9')
myData.add(entry)
})
vars.put('myData', new groovy.json.JsonBuilder(myData).toPrettyString())
log.info(vars.get('myData'))
The above example will generate 2 blocks:
If you want 10 - change 2 in the 1.upto(2, { line to 10
The generated data can be accessed as ${myData} where needed.
More information:
Apache Groovy - Parsing and producing JSON
Apache Groovy - Why and How You Should Use It

How to take any CSV file and convert it to JSON?(with python as a script engine) [Novice user trying to learn NiFi]

1) There is a CSV file containing the following information (the first row is the header):
first,second,third,total
1,4,9,14
7,5,2,14
3,8,7,18
2) I would like to find the sum of individual rows and generate a final file with a modified header. The final file should look like this:
[
{
"first": 1,
"second": 4,
"third": 9,
"total": 14
},
{
"first": 7,
"second": 5,
"third": 2,
"total": 14
},
{
"first": 3,
"second": 8,
"third": 7,
"total": 18
}
]
But it does not work and I am not sure how to fix this. Can anyone provide me an understanding on how to approach this problem?
NiFi flow:
Although i'm not into Python, by just googling around i think this might do it:
import csv
with open("YOURFILE.csv") as f:
reader = csv.DictReader(f)
data = [r for r in reader]
import json
with open('result.json', 'w') as outfile:
json.dump(data, outfile)
You can use Query Record processor and add new property as
total
select first,second,third,first+second+third total from FLOWFILE
Configure the CsvReader controller service with matching avro schema with int as datatype for all the fields and Json Setwriter controller service,Include total field name so that the output from Query Record processor will be all the columns and the sum of the columns as total.
Connect total relationship from Query Record processor for further processing
Refer to these links regarding Query Record and Configure Record Reader/Writer

BQ (Big Query) - Modify JSON output, to be accepted by BQ

Currently, using API to collect JSON file. I have managed to extract this output as I demonstrated below.
And now I'm on the stage, that I have JSON extraction and need to make in the way that BQ will accept it. Without too much manipulation ( as this output potentially will be loaded on the daily bases.
{
"stats": [{
"date": "2018-06-17T00:00:00.000Z",
"scores": {
"my-followers": 8113,
"my-listed": 15,
"my-favourites": 5289,
"my-followings": 230,
"my-statuses": 3107
}
}, {
"date": "2018-06-18T00:00:00.000Z",
"scores": {
"my-statuses": 3107,
"my-followings": 230,
"my-lost-followings": 0,
"my-new-followers": 0,
"my-new-statuses": 0,
"my-listed": 15,
"my-lost-followers": 5,
"my-followers": 8108,
"my-favourites": 5288,
"my-new-followings": 0
}
}
.....
],
"uid": "123456789"
}
ANy help will be appreciated.
Currently I have this error:
Errors:
query: Invalid field name "my-new-followings". Fields must contain only letters, numbers, and underscores, start with a letter or underscore, and be at most 128 characters long. Table: link_t_perf_test1_b58d4465_3a31_40cb_987f_9fb2d1de29dc_source (error code: invalidQuery
Even when "my-new-followings" contain only integer (up to 5 digit) number.

Reading a json file into a RDD (not dataFrame) using pyspark

I have the following file: test.json >
{
"id": 1,
"name": "A green door",
"price": 12.50,
"tags": ["home", "green"]
}
I want to load this file into a RDD. This is what I tried:
rddj = sc.textFile('test.json')
rdd_res = rddj.map(lambda x: json.loads(x))
I got an error:
Expecting object: line 1 column 1 (char 0)
I don't completely understand what does json.loads do.
How can I resolve this problem ?
textFile reads data line by line. Individual lines of your input are not syntactically valid JSON.
Just use json reader:
spark.read.json("test.json", multiLine=True)
or (not recommended) whole text files
sc.wholeTextFiles("test.json").values().map(json.loads)