I have data loaded from JSON and am trying to extract arbitrary nested values using a list as input, where the list corresponds to the names of successive children. I want a function get_value(data,lookup) that returns the value from data by treating each entry in lookup as a nested child.
In the example below, when lookup=['alldata','TimeSeries','rates'], the return value should be [1.3241,1.3233].
json_data = {'alldata':{'name':'CAD/USD','TimeSeries':{'dates':['2018-01-01','2018-01-02'],'rates':[1.3241,1.3233]}}}
def get_value(data,lookup):
res = data
for item in lookup:
res = res[item]
return res
lookup = ['alldata','TimeSeries','rates']
get_value(json_data,lookup)
My example works, but there are two problems:
It's inefficient - In my for loop, I copy the whole TimeSeries object to res, only to then replace it with the rates list. As #Andrej Kesely explained, res is a reference at each iteration, so data isn't being copied.
It's not concise - I was hoping to be able to find a concise (eg one or two line) way of extracting the data using something like list comprehension syntax
If you want one-liner and you are using Python 3.8, you can use assignment expression ("walrus operator"):
json_data = {'alldata':{'name':'CAD/USD','TimeSeries':{'dates':['2018-01-01','2018-01-02'],'rates':[1.3241,1.3233]}}}
def get_value(data,lookup):
return [data:=data[item] for item in lookup][-1]
lookup = ['alldata','TimeSeries','rates']
print( get_value(json_data,lookup) )
Prints:
[1.3241, 1.3233]
I don't think you can do it without a loop, but you could use a reducer here to increase readability.
functools.reduce(dict.get, lookup, json_data)
Related
Note: this is a similar but NOT duplicate question with How to use sqlx to query mysql IN a slice?. I'm asking for the Rust one.
This is what I try to do.
let v = vec![..];
sqlx::query("SELECT something FROM table WHERE column IN (?)").bind(v)
...
Then I got the following error
the trait bound `std::vec::Vec<u64>: sqlx::Encode<'_, _>` is not satisfied
Answer is in first on FAQ https://github.com/launchbadge/sqlx/blob/master/FAQ.md
How can I do a SELECT ... WHERE foo IN (...) query? In 0.6 SQLx will
support binding arrays as a comma-separated list for every database,
but unfortunately there's no general solution for that currently in
SQLx itself. You would need to manually generate the query, at which
point it cannot be used with the macros.
The error shows Vec is not an Encode that is required to be as a valid DB value. The Encode doc lists all the Rust types that have implemented the trait. Vec is not one.
You can use the following way to bind the parameters in IN with the values of a vector. Firstly, you need to expand the number of '?' in the IN expression to be the same number of the parameters. Then, you need to call bind to bind the values one by one.
let v = vec![1, 2];
let params = format!("?{}", ", ?".repeat(v.len()-1));
let query_str = format!("SELECT id FROM test_table WHERE id IN ( { } )", params);
let mut query = sqlx::query(&query_str);
for i in v {
query = query.bind(i);
}
let row = query.fetch_all(&pool).await?;
Please note if the target database is not MySql, you need to use $n, like $1, $2, instead of ?, as the parameter placeholder.
This is an extension to In Python, how to concisely get nested values in json data?
I have data loaded from JSON and am trying to replace arbitrary nested values using a list as input, where the list corresponds to the names of successive children. I want a function replace_value(data,lookup,value) that replaces the value in the data by treating each entry in lookup as a nested child.
Here is the structure of what I'm trying to do:
json_data = {'alldata':{'name':'CAD/USD','TimeSeries':{'dates':['2018-01-01','2018-01-02'],'rates':[1.3241,1.3233]}}}
def replace_value(data,lookup,value):
DEFINITION
lookup = ['alldata','TimeSeries','rates']
replace_value(json_data,lookup,[2,3])
# The following should return [2,3]
print(json_data['alldata']['TimeSeries']['rates'])
I was able to make a start with get_value(), but am stumped about how to do replacement. I'm not fixed to this code structure, but want to be able to programatically replace a value in the data given the list of successive children and the value to replace.
Note: it is possible that lookup can be of length 1
Follow the lookups until we're second from the end, then assign the value to the last lookup in the current object
def get_value(data,lookup): # Or whatever definition you like
res = data
for item in lookup:
res = res[item]
return res
def replace_value(data, lookup, value):
obj = get_value(data, lookup[:-1])
obj[lookup[-1]] = value
json_data = {'alldata':{'name':'CAD/USD','TimeSeries':{'dates':['2018-01-01','2018-01-02'],'rates':[1.3241,1.3233]}}}
lookup = ['alldata','TimeSeries','rates']
replace_value(json_data,lookup,[2,3])
print(json_data['alldata']['TimeSeries']['rates']) # [2, 3]
If you're worried about the list copy lookup[:-1], you can replace it with an iterator slice:
from itertools import islice
def replace_value(data, lookup, value):
it = iter(lookup)
slice = islice(it, len(lookup)-1)
obj = get_value(data, slice)
final = next(it)
obj[final] = value
You can obtain the parent to the final sub-dict first, so that you can reference it to alter the value of that sub-dict under the final key:
def replace_value(data, lookup, replacement):
*parents, key = lookup
for parent in parents:
data = data[parent]
data[key] = replacement
so that:
json_data = {'alldata':{'name':'CAD/USD','TimeSeries':{'dates':['2018-01-01','2018-01-02'],'rates':[1.3241,1.3233]}}}
lookup = ['alldata','TimeSeries','rates']
replace_value(json_data,lookup,[2,3])
print(json_data['alldata']['TimeSeries']['rates'])
outputs:
[2, 3]
Once you have get_value
get_value(json_data, lookup[:-1])[lookup[-1]] = value
I have a script that scans a DynamoDB table that stores my instance IDs. Then I try to query another table to see if it also has that same instance and get all of the metadata attributes in a master table. When I iterate through the query using the instance ID from the initial scan of the first table, I am noticing each character of the instance id string is being printed to a new line, instead of the entire string on one line. I am confused how to fix this. Below is my code, sample output, and the expected output.
CODE:
import boto3
import json
from boto3.dynamodb.conditions import Key, Attr
def table_diff():
dynamo = boto3.client('dynamodb')
dynamodb = boto3.resource('dynamodb')
table_missing = dynamodb.Table('RunningInstances')
missing_response = dynamo.scan(TableName='CWPMissingAgent')
for instances in missing_response['Items']:
instance_id = instances['missing_instances']['S']
# This works how I want, prints i-xxxxx
print(instance_id)
for id in instance_id:
# This does not print how I want (vertically)
print(id)
query_response = table_missing.query(KeyConditionExpression=Key('ID').eq(id))
OUTPUT:
i
-
x
x
x
x
x
EXPECTED OUTPUT:
i-xxxxx
etc etc
instance_id is a string. Thus, when you loop over it (for id in instance_id), you are actually looping over each character in the string, and printing them out individually.
Why do you try to loop over it, when you say that just printing it produces the correct result?
I have written a loop to retrieve data from an API (Limesurvey) based on a list of ids and fill a row of a dataframe with the result of each loop.
I have a list with ids like this:
# list of ids
ids = ['1','427',... ,'847']
My code to query the API based on each item of the list looks as follows:
method = "get_participant_properties"
params = OrderedDict([
("sSessionKey", api.session_key),
("iSurveyID", 12345),
("aTokenQueryProperties", t),
])
# loop through API query with the 'aTokenQueryProperties' stored in the list 'tids'.
attributes = []
for t in ids:
attributes.append(api.query(method=method, params=params))
pd.DataFrame(attributes)
Unfortunately, the result is a dataframe with 158 rows, and each row is the same, i.e. the query result of the last id in my list (847).
You are not passing in the t from the loop. The t in the params definition is unrelated; if I were to run your code now I'd get a NameError exception because t hasn't been set at that point. The t expression in the params mapping is not live, it is not updated each loop iteration.
Set the 'aTokenQueryProperties' key in the loop:
method = "get_participant_properties"
params = OrderedDict([
("sSessionKey", api.session_key),
("iSurveyID", 12345),
("aTokenQueryProperties", None),
])
attributes = []
for t in ids:
params["aTokenQueryProperties"] = t
attributes.append(api.query(method=method, params=params))
Setting "aTokenQueryProperties" to None in the params OrderedDict object at the start is optional; you'd only need to do this if reserving its exact location in the params order is important and even then, because in your example it is the last element in the mapping, you'd end up with the same output anyway.
How to convert a msg.payload like 12345,67890 into "12345","67890" ?
Basically I receive a payload with many values and need to pass them to a mysql node.
Thanks in advance.
Sep
Depending on the exact type of input there are a couple options
First the core CSV node will split a payload into either an array or an object (if the first line of multiple lines contains the column names).
Second if you just have a single input then you can break it up with a function node using the normal NodeJS/Javascript functions for working with Strings. e.g.
var chunks = msg.payload.split(',');
msg.payload = {};
msg.paylaod.first = chunks[0];
msg.payload.second = chunk[1];
return msg;