I am quite new at this. I have a Celery- FastAPI model that takes JSON Tasks and basically does some easy calculations with them.
#app.post("/ex1")
def run_task(data=Body(...)):
a = int(data["a"])
b = data["b"]
add = data["add"]
sub = data["sub"]
mul = data["mul"]
div = data["div"]
task = addieren_task.delay(a, b, add, sub, mul, div)
return JSONResponse({"Result": task.get()})
This is how the transition of the JSON to the task looks like.
I send the tasks with bash with this command
curl http://localhost:8000/ex1 -H "Content-Type: application/json" --data '{"a": 4, "b":19, "add": 0, "sub": 1, "mul": 0, "div": 1}'
Can someone help me send multiple JSON tasks so I can monitor the whole model better using Flower ?
Thanks in advance
Related
I have the following task:
this_is_a_task = SimpleHttpOperator(
task_id= 'task_id',
method='POST',
http_conn_id='conn_id',
endpoint='/?test=foo',
# data={"test": "foo"},
headers={"Content-Type": "application/json"}
on the cloud functions side, I'm trying to catch the parameters with the two following ways:
# catching data
# test_data = request.get_json().get('test')
# print('test: {}'.format(test))
# catching end point
test_endpoint = request.args.get('test')
print('test: {}'.format(test))
the second option is working (request.args.get('test')) however when trying the first option request.get_json().get('test') I'm getting a 400 request error.
so if I'm not using the endpoint variable from my SimpleHttpOperator how can I catch the json object pass into the data variable?
I've tried to replicate your issue and based on this documentation you need to add json.dumps when you are calling a POST with json data. Then provide authentication credentials as a Google-generated ID token stored in an Authorization header.
See below sample code:
import datetime
import json
from airflow import models
from airflow.operators import bash
from airflow.providers.http.operators.http import SimpleHttpOperator
YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)
default_args = {
'owner': 'Composer Example',
'depends_on_past': False,
'email': [''],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': datetime.timedelta(minutes=5),
'start_date': YESTERDAY,
}
with models.DAG(
'composer_quickstart',
catchup=False,
default_args=default_args,
schedule_interval=datetime.timedelta(days=1)) as dag:
# Print the dag_run id from the Airflow logs
gen_auth = bash.BashOperator(
task_id='gen_auth', bash_command='gcloud auth print-identity-token '
)
auth_token = "{{ task_instance.xcom_pull(task_ids='gen_auth') }}"
this_is_a_task = SimpleHttpOperator(
task_id='task_id',
method='POST',
http_conn_id='cf_conn1',
data=json.dumps({"test": "foo"}),
headers={"Content-Type": "application/json","Authorization": "Bearer " + auth_token}
)
gen_auth >> this_is_a_task
On the cloud functions side tried to use below sample code:
test_data = request.get_json().get('test')
print(test_data)
return test_data
You can also test your function using this curl command:
curl -i -X POST -H "Content-Type:application/json" -d '{"test": "foo"}' "Authorization: bearer $(gcloud auth print-identity-token)" https://function-5-k6ssrsqwma-uc.a.run.app
i am pretty new to python and i tried some things but can't seem to find the solution for this.
I am making a get request to a ldap local server and the response is a list, i don't know if this is a list of lists or just a list.
[
"cn=Philip J. Fry,ou=people,dc=planetexpress,dc=com",
"cn=Turanga Leela,ou=people,dc=planetexpress,dc=com",
"cn=Bender Bending Rodr\u00edguez,ou=people,dc=planetexpress,dc=com"
]
I want to return a json with only the "cn" values.
Like this:
[
"Philip",
"Turanga",
"Bender"
]
Thanks for the help
I would convert the strings into dictionaries:
def ldap_to_dict(string):
d = {}
for pair in string.split(","):
key, val = pair.split("=")
if key in d: # handle multiple value case
if isinstance(d[key], list):
d[key].append(val)
else:
d[key] = [d[key], val]
else:
d[key] = val
return d
names = [ldap_to_dict(ldap_string)["cn"] for ldap_string in original_list]
Aftermath:
>>> names
['Philip J. Fry', 'Turanga Leela', 'Bender Bending RodrÃguez']
Thanks guys for the quick responses.
I manage to do it like this:
list1 = []
list2 = []
list3 = []
#app.route('/users', methods=['GET'])
def list_users():
string = (ldap.get_group_members('ship_crew'))
for user in string:
list1.append(user.split(",")[0])
a = list1
for user in a:
list2.append(user.split("=")[1])
b = list2
for user in b:
list3.append(user.split(" ")[0])
return jsonify(list3)
Output:
['Philip', 'Turanga', 'Bender']
But now when i'am trying to return it as json with this command:
curl -X GET -H "Content-type: application/json charset=UTF-8" -H "Accept: application/json" http://localhost:5000/users
I get:
TypeError: a bytes-like object is required, not 'str'
Even if i json.dump my list3 and get:
["Philip", "Turanga", "Bender"]
I got the same error.
I'am getting my string from a ldap.
What's the problem here?
Thanks
I have been experimenting with Plumber in R recently, and am having success when I pass the following data using a POST request;
{"Gender": "F", "State": "AZ"}
This allows me to write a function like the following to return the data.
#* #post /score
score <- function(Gender, State){
data <- list(
Gender = as.factor(Gender)
, State = as.factor(State))
return(data)
}
However, when I try to POST an array of JSON objects, I can't seem to access the data through the function
[{"Gender":"F","State":"AZ"},{"Gender":"F","State":"NY"},{"Gender":"M","State":"DC"}]
I get the following error
{
"error": [
"500 - Internal server error"
],
"message": [
"Error in is.factor(x): argument \"Gender\" is missing, with no default\n"
]
}
Does anyone have an idea of how Plumber parses JSON? I'm not sure how to access and assign the fields to vectors to score the data.
Thanks in advance
I see two possible solutions here. The first would be a command line based approach which I assume you were attempting. I tested this on a Windows OS and used column based data.frame encoding which I prefer due to shorter JSON string lengths. Make sure to escape quotation marks correctly to avoid 'argument "..." is missing, with no default' errors:
curl -H "Content-Type: application/json" --data "{\"Gender\":[\"F\",\"F\",\"M\"],\"State\":[\"AZ\",\"NY\",\"DC\"]}" http://localhost:8000/score
# [["F","F","M"],["AZ","NY","DC"]]
The second approach is R native and has the advantage of having everything in one place:
library(jsonlite)
library(httr)
## sample data
lst = list(
Gender = c("F", "F", "M")
, State = c("AZ", "NY", "DC")
)
## jsonify
jsn = lapply(
lst
, toJSON
)
## query
request = POST(
url = "http://localhost:8000/score?"
, query = jsn # values must be length 1
)
response = content(
request
, as = "text"
, encoding = "UTF-8"
)
fromJSON(
response
)
# [,1]
# [1,] "[\"F\",\"F\",\"M\"]"
# [2,] "[\"AZ\",\"NY\",\"DC\"]"
Be aware that httr::POST() expects a list of length-1 values as query input, so the array data should be jsonified beforehand. If you want to avoid the additional package imports altogether, some system(), sprintf(), etc. magic should do the trick.
Finally, here is my plumber endpoint (living in R/plumber.R and condensed a little bit):
#* #post /score
score = function(Gender, State){
lapply(
list(Gender, State)
, as.factor
)
}
and code to fire up the API:
pr = plumber::plumb("R/plumber.R")
pr$run(port = 8000)
I am using mosquitto version 1.4.10 with tls-certificate. I am using this plugin https://github.com/mbachry/mosquitto_pyauth to authorize a user.And it works well for mosquitto_pub ( as in, when someone tries to publish , it gets authorized by the module first ).
However, it seems that mosquitto_sub is able to subscribe to anything without authorizing. How do I force security when someone is just trying to access a topic in read only mode?
I went through the mosquitto.conf file and cant seem to find anything related to this.
for example, I am able to subscribe like this:
mosquitto_sub --cafile /etc/mosquitto/ca.crt --cert /etc/mosquitto/client.crt --key /etc/mosquitto/client.key -h ubuntu -p 1883 -t c/# -d
and able to see messages coming from some publisher like this:
mosquitto_pub --cafile /etc/mosquitto/ca.crt --cert /etc/mosquitto/client.crt --key /etc/mosquitto/client.key -h ubuntu -p 1883 -t c/2/b/3/p/3/rt/13/r/123 -m 32 -q 1
What I am trying to do is prevent mosquitto_sub reading all messages at root level without authorization .
the python code that does the authorization looks like this : ( auth data is stored in cassandra db )
import sys
import mosquitto_auth
from cassandra.cluster import Cluster
from cassandra import ConsistencyLevel
## program entry point from mosquitto...
def plugin_init(opts):
global cluster, session, select_device_query
conf = dict(opts)
cluster = Cluster(['192.168.56.102'])
session = cluster.connect('hub')
select_device_query = session.prepare('SELECT * from devices where uid=?')
select_device_query.consistency_level = ConsistencyLevel.QUORUM
print 'Cassandra cluster initialized'
def acl_check(clientid, username, topic, access):
device_data = session.execute(select_device_query, [username])
if device_data.current_rows.__len__() > 0:
device_data = device_data[0]
# sample device data looks like this :
# Row(uid=u'08:00:27:aa:8f:91', brand=3, company=2, device=15617, property=3, room=490, room_number=u'3511', room_type=13, stamp=datetime.datetime(2016, 12, 12, 6, 29, 54, 723000))
subscribable_topic = 'c/' + str(device_data.company) \
+ '/b/' + str(device_data.brand) \
+ '/p/' + str(device_data.property) \
+ '/rt/' + str(device_data.room_type) \
+ '/r/' + str(device_data.room) \
+ '/#'
matches = mosquitto_auth.topic_matches_sub(subscribable_topic, topic)
print 'ACL: user=%s topic=%s, matches = %s' % (username, topic, matches)
return matches
return False
function acl_check seems to be always called when mosquitto_pub tries to connect, but never called when mosquitto_sub connects.
the C code behind this python module is here: https://github.com/mbachry/mosquitto_pyauth/blob/master/auth_plugin_pyauth.c
add the following to your mosquitto.conf
...
allow_anonymous false
...
This will stop users without credential from logging on to the broker.
You can also add an acl rule for the anonymous user if there are certain topics you would want unauthenticated clients to be able to see.
As per my understanding "ffprobe" will provide file related data in JSON format. So, I have installed the ffprobe in my Ubuntu machine but I don't know how to access the ffprobe JSON response using Java/Grails.
Expected response format:
{
"format": {
"filename": "/Users/karthick/Documents/videos/TestVideos/sample.ts",
"nb_streams": 2,
"nb_programs": 1,
"format_name": "mpegts",
"format_long_name": "MPEG-TS (MPEG-2 Transport Stream)",
"start_time": "1.430800",
"duration": "170.097489",
"size": "80425836",
"bit_rate": "3782576",
"probe_score": 100
}
}
This is my groovy code
def process = "ffprobe -v quiet -print_format json -show_format -show_streams HelloWorld.mpeg ".execute()
println "Found ${process.text}"
render process as JSON
I m able to get the process object and i m not able to get the json response
Should i want to convert the process object to json object?
OUTPUT:
Found java.lang.UNIXProcess#75566697
org.codehaus.groovy.grails.web.converters.exceptions.ConverterException: Error converting Bean with class java.lang.UNIXProcess
Grails has nothing to do with this. Groovy can execute arbitrary shell commands in a very simplistic way:
"mkdir foo".execute()
Or for more advanced features, you might look into using ProcessBuilder. At the end of the day, you need to execute ffprobe and then capture the output stream of JSON to use in your app.
Groovy provides a simple way to execute command line processes. Simply
write the command line as a string and call the execute() method.
The execute() method returns a java.lang.Process instance.
println "ffprobe <options>".execute().text
[Source]