Currently, I'm doing a currency converter bot on telegram using python 3.
def on_callback_query(msg):
query_id, from_id, query_data = telepot.glance(msg, flavor='callback_query')
print('Callback Query:', query_id, from_id, query_data)
url = "http://api.fixer.io/latest?base=SGD&symbols="
symbols = query_data
response = requests.get(url + symbols)
data = response.json()
print(data)
if(query_data == 'SGD'):
bot.sendMessage(from_id, data)
elif (query_data == 'EUR'):
bot.sendMessage(from_id, data)
The data here will out {'rates': {'EUR': 0.62177}, 'base': 'SGD', 'date': '2017-09-18'} in my telegram chat. Is there anyway to beautify the output, i.e. take out the curly brackets and quotes?
This format named JSON and you can parse it as an object and simply print it with a custom format.
See this.
The result is in JSON format which is then converted to a dictionary, you can extract each element from the dictionary and use it to create a better format.
You can write in any format you like, here's an example
if(query_data == 'SGD'):
result = "Rates in EUR: {}\nBase: {}\nDate: {}".format(data['rates']['EUR'], data['base'], data['date'])
bot.sendMessage(from_id, result)
if(query_data == 'EUR'):
result = "Rates in SGD: {}\nBase: {}\nDate: {}".format(data['rates']['SGD'], data['base'], data['date'])
bot.sendMessage(from_id, result)
There's almost definitely a more efficient way of doing this than the way I did it, but it works.
str(output).replace("{","").replace("}","").replace("'","")
Related
this is my second try to explain a bit more precisely what I'm looking for ;-)
I set a webhook in Mailchimp that fires every time a new subscriber of an audience appears. Mailchimp sends a HTTP POST request to a Jira Sriptrunner REST endpoint.
The content type of this request is application/x-www-form-urlencoded.
Within the Jira endpoint I would like to read the request data. How can I do that?
The payload (raw body) I receive looks like this:
type=unsubscribe&fired_at=2020-05-26+07%3A04%3A42&data%5Baction%5D=unsub&data%5Breason%5D=manual&data%5Bid%5D=34f28a4516&data%5Bemail%5D=examlple%40bla.com&data%5Bemail_type%5D=html&data%5Bip_opt%5D=xx.xxx.xxx.198&data%5Bweb_id%5D=118321378&data%5Bmerges%5D%5BEMAIL%5D=example%40bla.com&data%5Bmerges%5D%5BFNAME%5D=Horst&data%5Bmerges%5D%5BLNAME%5D=Schlemmer&data%5Bmerges%5D%5BCOMPANY%5D=First&data%5Bmerges%5D%5BADDRESS%5D%5Baddr1%5D=XXX
Now I would like to parse the data of the raw body into a JSON or something similiar.
The result might look like this:
{
"web_id": 123,
"email": "example#bla.com",
"company": "First",
...
}
Meanwhile I searched around a little and found something like the node.js "querystring" module. It would be great if there is something similiar within Groovy or any other way to parse the data of application/x-www-form-urlencoded to json format.
Best regards and thanks in advance
Bernhard
def body = "type=unsubscribe&fired_at=2020-05-26+07%3A04%3A42&data%5Baction%5D=unsub&data%5Breason%5D=manual&data%5Bid%5D=34f28a4516&data%5Bemail%5D=examlple%40bla.com&data%5Bemail_type%5D=html&data%5Bip_opt%5D=xx.xxx.xxx.198&data%5Bweb_id%5D=118321378&data%5Bmerges%5D%5BEMAIL%5D=example%40bla.com&data%5Bmerges%5D%5BFNAME%5D=Horst&data%5Bmerges%5D%5BLNAME%5D=Schlemmer&data%5Bmerges%5D%5BCOMPANY%5D=First&data%5Bmerges%5D%5BADDRESS%5D%5Baddr1%5D=XXX"
def map = body.split('&').collectEntries{e->
e.split('=').collect{ URLDecoder.decode(it, "UTF-8") }
}
assert map.'data[merges][EMAIL]'=='example#bla.com'
map.each{println it}
prints:
type=unsubscribe
fired_at=2020-05-26 07:04:42
data[action]=unsub
data[reason]=manual
data[id]=34f28a4516
data[email]=examlple#bla.com
data[email_type]=html
data[ip_opt]=xx.xxx.xxx.198
data[web_id]=118321378
data[merges][EMAIL]=example#bla.com
data[merges][FNAME]=Horst
data[merges][LNAME]=Schlemmer
data[merges][COMPANY]=First
data[merges][ADDRESS][addr1]=XXX
A imple no-brainer groovy:
def a = '''
data[email_type]: html
data[web_id]: 123
fired_at: 2020-05-26 07:28:25
data[email]: example#bla.com
data[merges][COMPANY]: First
data[merges][FNAME]: Horst
data[ip_opt]: xx.xxx.xxx.xxx
data[merges][PHONE]: xxxxx
data[merges][ADDRESS][zip]: 33615
type: subscribe
data[list_id]: xxXXyyXX
data[merges][ADDRESS][addr1]: xxx.xxx'''
def res = [:]
a.eachLine{
def parts = it.split( /\s*:\s*/, 2 )
if( 2 != parts.size() ) return
def ( k, v ) = parts
def complexKey = ( k =~ /\[(\w+)\]/ ).findAll()
if( complexKey ) complexKey = complexKey.last().last()
res[ ( complexKey ?: k ).toLowerCase() ] = v
}
res
gives:
[email_type:html, web_id:123, fired_at:2020-05-26 07:28:25,
email:example#bla.com, company:First, fname:Horst, ip_opt:xx.xxx.xxx.xxx,
phone:xxxxx, zip:33615, type:subscribe, list_id:xxXXyyXX, addr1:xxx.xxx]
I found a solution finally. I hope you understand and maybe it helps others too ;-)
Starting from daggett's answer I did the following:
// Split body and remove unnecessary characters
def map = body.split('&').collectEntries{e->
e.split('=').collect{ URLDecoder.decode(it, "UTF-8") }
}
// Processing the map to readable stuff
def prettyMap = new JsonBuilder(map).toPrettyString()
// Convert the pretty map into a json object
def slurper = new JsonSlurper()
def jsonObject = slurper.parseText(prettyMap)
(The map looks pretty much like in daggett's answer.
prettyMap)
Then I extract the keys:
// Finally extracting customer data
def type = jsonObject['type']
And I get the data I need. For example
Type : subscribe
...
First Name : Heinz
...
Thanks to daggett!
I have searched what I can and I don't seem to be finding the answer I need. Granted I may not be wording it properly. I have tried using .find or even .rindex to count backwards, but no such luck. The value I receive from the JSON looks something like this:
"AdditionalData":"<Data><Entry Key=\"utm_campaign\" Value=\"j2c\" />
<Entry Key=\"utm_medium\" Value=\"cpc\" /><Entry Key=\"utm_source\"
Value=\"j2c\" /><Entry Key=\"job_id\" Value=\"300_xxxx_10703\" /></Data>"
I need to be able to grab the value for the key "job_id", so the "300_xxxx_11233". This value will change per object returned by the JSON response. Any help would be appreciated, and please let me know if this is already out there and I just missed it.
If the response format remains the same with every request, you could use a plain regexp expression to fetch your data, even without parsing JSON. Example:
response = "<Data><Entry Key=\"utm_campaign\" Value=\"j2c\" /><Entry Key=\"utm_medium\" Value=\"cpc\" /><Entry Key=\"utm_source\" Value=\"j2c\" /><Entry Key=\"job_id\" Value=\"300_xxxx_10703\" /></Data>"
match = response.match(%r{job_id\\?"\s+Value=\\?"(.+)\\?"}i)
match[1] if match # => "300_xxxx_10703"
If the response format can change (for example, if the order of the attributes of Entry element can change), then you need to parse JSON and use some HTML parser, such as Nokigiri, to fetch required attrbute. Code example:
parsed_response = JSON.parse(response)
doc = Nokogiri::HTML(parsed_response['AdditionalData'])
job_id = nil
doc.css('Entry').each do |el|
if el['Key'] == 'job_id'
job_id = el['Value']
break
end
end
I have generated a JSON file from data source which is of the format.
{}{}{}
I wish to convert this format to comma separated JSON Array as. [{},{},{}].
End goal is to push the JSON data [{},{},{}] to MongoDB.
My pythoin solution (although naive) looks something like this:
def CreateJSONArrayFile(filename):
print('Opening file with JSON data')
with open(filename) as data_file:
raw_data = data_file.read()
tweaked_data = raw_data.replace('}{', '}^|{')
split_data = tweaked_data.split('^|')
outfile = open('split_data.json', 'w')
outfile.write('[')
for item in split_data:
outfile.write("%s," % item)
outfile.write(']')
print('split_data.json Created with JSON Array')
The above code is giving me wrong results.
Can you please help me optimize the solution? Please let me know if you need more details from my end.
I'm with davedwards on this one, but if not an option -- I think this gets you what you are after.
myJson = """{"This": "is", "a": "test"} {"Of": "The", "Emergency":"Broadcast"}"""
myJson = myJson.replace("} {", "}###{")
new_list = myJson.split('###')
print(new_list)
yields:
['{"This": "is", "a": "test"}', '{"Of": "The", "Emergency":"Broadcast"}']
Not saying it is the most elegant way : )
I have this massive json file (8gb), and I run out of memory when trying to read it in to Python. How would I implement a similar procedure using ijson or some other library that is more efficient with large json files?
import pandas as pd
#There are (say) 1m objects - each is its json object - within in this file.
with open('my_file.json') as json_file:
data = json_file.readlines()
#So I take a list of these json objects
list_of_objs = [obj for obj in data]
#But I only want about 200 of the json objects
desired_data = [obj for obj in list_of_objs if object['feature']=="desired_feature"]
How would I implement this using ijson or something similar? Is there a way I can extract the objects I want without reading in the whole JSON file?
The file is a list of objects like:
{
"review_id": "zdSx_SD6obEhz9VrW9uAWA",
"user_id": "Ha3iJu77CxlrFm-vQRs_8g",
"business_id": "tnhfDv5Il8EaGSXZGiuQGg",
"stars": 4,
"date": "2016-03-09",
"text": "Great place to hang out after work: the prices are decent, and the ambience is fun. It's a bit loud, but very lively. The staff is friendly, and the food is good. They have a good selection of drinks.",
"useful": 0,
"funny": 0,
}
The file is a list of objects
This is a little ambiguous. Looking at your code snippet it looks like your file contains separate JSON object on each line. Which is not the same as the actual JSON array that starts with [, ends with ] and has , between items.
In the case of a json-per-line file it's as easy as:
import json
from itertools import islice
with(open(filename)) as f:
objects = (json.loads(line) for line in f)
objects = islice(objects, 200)
Note the differences:
you don't need .readlines(), the file object itself is an iterable that yields individual lines
parentheses (..) instead of brackets [..] in (... for line in f) create a lazy generator expression instead of a Python list in memory with all the lines
islice(objects, 200) will give you the first 200 items without iterating further. If objects would've been a list you could just do objects[:200]
Now, if your file is actually a JSON array then you indeed need ijson:
import ijson # or choose a faster backend if needed
from itertools import islice
with open(filename) as f:
objects = ijson.items(f, 'item')
objects = islice(objects, 200)
ijson.items returns a lazy iterator over a parsed array. The 'item' in the second parameter means "each item in a top-level array".
The problem is that not all JSON comes nicely formatted and you cannot rely on line-by-line parsing to extract your objects.
I understood your "acceptance criteria" as "want to collect only those JSON objects whose specified keys contain specified values". For example, only collecting objects about a person if that person's name is "Bob". The following function will provide a list of all objects that fit your criteria. Parsing is done character by character (something that would be much more efficient in C, but Python is still pretty good). This should be more robust because it doesn't care about newlines, formatting etc. I tested this on both formatted and unformatted JSON with 1,000,000 objects.
import json
def parse_out_objects(file, feature, desired_value):
with open(file) as f:
compose_object_flag = False
ignore_characters_flag = False
object_string = ''
selected_objects = []
json_object = None
while True:
c = f.read(1)
if c == '"':
ignore_characters_flag = not ignore_characters_flag
if c == '{' and ignore_characters_flag == False:
compose_object_flag = True
if c == '}' and compose_object_flag == True and ignore_characters_flag == False:
compose_object_flag = False
object_string = object_string + '}'
json_object = json.loads(object_string)
if json_object[feature] == desired_value:
selected_objects.append(json_object)
object_string = ''
if compose_object_flag == True:
object_string = object_string + c
if not c:
break
return selected_objects
I am using Dell Boomi to map data from one system to another. I can use groovy in the maps but have no experience with it. I tried to do this with the other Boomi tools, but have been told that I'll need to use groovy in a script. My inbound data is:
132265,Brown
132265,Gold
132265,Gray
132265,Green
I would like to output:
132265,"Brown,Gold,Gray,Green"
Hopefully this makes sense! Any ideas on the groovy code to make this work?
It can be elegantly solved with groupBy and the spread operator:
#Grapes(
#Grab(group='org.apache.commons', module='commons-csv', version='1.2')
)
import org.apache.commons.csv.*
def csv = '''
132265,Brown
132265,Gold
132265,Gray
132265,Green
'''
def parsed = CSVParser.parse(csv, CSVFormat.DEFAULT.withHeader('code', 'color')
parsed.records.groupBy({ it.code }).each { k,v -> println "$k,\"${v*.color.join(',')}\"" }
The above prints:
132265,"Brown,Gold,Gray,Green"
Well, I don't know how are you getting your data, but here is a general way to achieve your goal. You can use a library, such as the one bellow to parse the csv.
https://github.com/xlson/groovycsv
The example for your data would be:
#Grab('com.xlson.groovycsv:groovycsv:1.1')
import static com.xlson.groovycsv.CsvParser.parseCsv
def csv = '''
132265,Brown
132265,Gold
132265,Gray
132265,Green
'''
def data = parseCsv(csv)
I believe you want to associate the number with various values of colors. So for each line you can create a map of the number and the colors associated with that number, splitting the line by ",":
map = [:]
for(line in data) {
number = line.split(',')[0]
colour = line.split(',')[1]
if(!map[number])
map[number] = []
map[number].add(colour)
}
println map
So map should contain:
[132265:["Brown","Gold","Gray","Green"]]
Well, if it is not what you want, you can extract the general idea.
Assuming your data is coming in as a comma separated string of data like this:
"132265,Brown 132265,Gold 132265,Gray 132265,Green 122222,Red 122222,White"
The following Groovy script code should do the trick.
def csvString = "132265,Brown 132265,Gold 132265,Gray 132265,Green 122222,Red 122222,White"
LinkedHashMap.metaClass.multiPut << { key, value ->
delegate[key] = delegate[key] ?: []; delegate[key] += value
}
def map = [:]
def csv = csvString.split().collect{ entry -> entry.split(",") }
csv.each{ entry -> map.multiPut(entry[0], entry[1]) }
def result = map.collect{ k, v -> k + ',"' + v.join(",") + '"'}.join("\n")
println result
Would print:
132265,"Brown,Gold,Gray,Green"
122222,"Red,White"
Do you HAVE to use scripting for some reason? This can be easily accomplished with out-of-the-box Boomi functionality.
Create a map function that prepends the ID field to a string of your choice (i.e. 222_concat_fields). Then use that value to set a dynamic process prop with that value.
The value of the process prop will contain the result of concatenating the name fields. Simply adding this function to your map should take care of it. Then use the final value to populate your result.
Well it depends upon the data how is it coming.
If the data which you have posted in the question is coming in a single document, then you can easily handle this in a map with groovy scripting.
If the data which you have posted in the question is coming into multiple documents i.e.
doc1: 132265,Brown
doc2: 132265,Gold
doc3: 132265,Gray
doc4: 132265,Green
In that case it cannot be handled into map. You will need to use Data Process Step with Custom Scripting.
For the code which you are asking to create in groovy depends upon the input profile in which you are getting the data. Please provide more information i.e. input profile, fields etc.