How to schedule a downtime in icinga2 by using icinga-api with groovy? - json

I'm searching for a way to schedule a downtime in icinga2 with a groovy script.
I already tried creating a small groovy script. Tried using the examples from icinga documentation:
curl -u root:icinga -k -s 'https://localhost:5665/v1/actions/schedule-downtime?type=Host&filter=host.vars.os==%22Linux%22' -d '{ "author" : "michi", "comment": "Maintenance.", "start_time": 1441136260, "end_time": 1441137260, "duration": 1000 }' -X POST | python -m json.tool
but adapting this to my script didn't work. Very important are the " around each attribute name, I noted.

Solution was this way:
Using wslite as webservice client. This is the minimal example.
Now I connect to my server with api enabled. The certificate is self signed, why "sslTrustAllCerts" was needed.
I select all services from my host "testserver" and set the downtime (duration in seconds).
#Grab('com.github.groovy-wslite:groovy-wslite:1.1.2')
import wslite.rest.*
import wslite.http.auth.*
def client = new RESTClient("https://myicinga2server:5665/")
client.authorization = new HTTPBasicAuthorization("root", "secret")
def timeFrom = System.currentTimeMillis() / 1000L
def timeDurationSec = 600
def timeTo = timeFrom + timeDurationSec
try
{
def response = client.post(
path: '/v1/actions/schedule-downtime?type=Service&filter=host.name==%22testserver%22',
headers: ["Accept": "application/json"],
sslTrustAllCerts: true)
{
json "author":"mstein", "comment":"Test-Downtime", "start_time": timeFrom, "end_time": timeTo, "duration": timeDurationSec, "fixed": true
}
assert 200 == response.statusCode
print response.text
}
catch (Exception exc)
{
println "Error: " + exc.getClass().toString()
println "Message: " + exc.getMessage()
println "Response: " + exc.getResponse()
System.exit(1)
}
That worked for me!

Related

pass data with SimpleHttpOperator to trigger cloud function 2nd gen

I have the following task:
this_is_a_task = SimpleHttpOperator(
task_id= 'task_id',
method='POST',
http_conn_id='conn_id',
endpoint='/?test=foo',
# data={"test": "foo"},
headers={"Content-Type": "application/json"}
on the cloud functions side, I'm trying to catch the parameters with the two following ways:
# catching data
# test_data = request.get_json().get('test')
# print('test: {}'.format(test))
# catching end point
test_endpoint = request.args.get('test')
print('test: {}'.format(test))
the second option is working (request.args.get('test')) however when trying the first option request.get_json().get('test') I'm getting a 400 request error.
so if I'm not using the endpoint variable from my SimpleHttpOperator how can I catch the json object pass into the data variable?
I've tried to replicate your issue and based on this documentation you need to add json.dumps when you are calling a POST with json data. Then provide authentication credentials as a Google-generated ID token stored in an Authorization header.
See below sample code:
import datetime
import json
from airflow import models
from airflow.operators import bash
from airflow.providers.http.operators.http import SimpleHttpOperator
YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)
default_args = {
'owner': 'Composer Example',
'depends_on_past': False,
'email': [''],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': datetime.timedelta(minutes=5),
'start_date': YESTERDAY,
}
with models.DAG(
'composer_quickstart',
catchup=False,
default_args=default_args,
schedule_interval=datetime.timedelta(days=1)) as dag:
# Print the dag_run id from the Airflow logs
gen_auth = bash.BashOperator(
task_id='gen_auth', bash_command='gcloud auth print-identity-token '
)
auth_token = "{{ task_instance.xcom_pull(task_ids='gen_auth') }}"
this_is_a_task = SimpleHttpOperator(
task_id='task_id',
method='POST',
http_conn_id='cf_conn1',
data=json.dumps({"test": "foo"}),
headers={"Content-Type": "application/json","Authorization": "Bearer " + auth_token}
)
gen_auth >> this_is_a_task
On the cloud functions side tried to use below sample code:
test_data = request.get_json().get('test')
print(test_data)
return test_data
You can also test your function using this curl command:
curl -i -X POST -H "Content-Type:application/json" -d '{"test": "foo"}' "Authorization: bearer $(gcloud auth print-identity-token)" https://function-5-k6ssrsqwma-uc.a.run.app

How to send json message body in aws SNS using console

I am doing a hands on where I want to add an SNS trigger to a lambda function which then sends a message to a slack channel. There is a blueprint for this lambda in python and also a template test event which looks like the following
{
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "arn:aws:sns:EXAMPLE",
"EventSource": "aws:sns",
"Sns": {
"SignatureVersion": "1",
"Timestamp": "1970-01-01T00:00:00.000Z",
"Signature": "EXAMPLE",
"SigningCertUrl": "EXAMPLE",
"MessageId": "12345",
"Message": {
"AlarmName": "SlackAlarm",
"NewStateValue": "OK",
"NewStateReason": "Threshold Crossed: 1 datapoint (0.0) was not greater than or equal to the threshold (1.0)."
},
"MessageAttributes": {
"Test": {
"Type": "String",
"Value": "TestString"
},
"TestBinary": {
"Type": "Binary",
"Value": "TestBinary"
}
},
"Type": "Notification",
"UnsubscribeUrl": "EXAMPLE",
"TopicArn": "arn:aws:sns:EXAMPLE",
"Subject": "TestInvoke"
}
}
]
The code in lambda handler from the blueprint is as follows
import boto3
import json
import logging
import os
from base64 import b64decode
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
HOOK_URL = os.environ['kmsEncryptedHookUrl']
SLACK_CHANNEL = os.environ['slackChannel']
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info("Event: " + str(event))
message = event['Records'][0]['Sns']['Message']
logger.info("Message: " + str(message))
alarm_name = message['AlarmName']
new_state = message['NewStateValue']
reason = message['NewStateReason']
slack_message = {
'channel': SLACK_CHANNEL,
'text': "%s state is now %s: %s" % (alarm_name, new_state, reason)
}
req = Request(HOOK_URL, json.dumps(slack_message).encode('utf-8'))
try:
response = urlopen(req)
response.read()
logger.info("Message posted to %s", slack_message['channel'])
except HTTPError as e:
logger.error("Request failed: %d %s", e.code, e.reason)
except URLError as e:
logger.error("Server connection failed: %s", e.reason)
When I run the test event, the lambda runs successfully.
I wanted to publish a message in SNS topic from the console to see if the lambda is triggered correctly. But when I try to publish the JSON object as a message body, I am getting the error
[ERROR] TypeError: string indices must be integersTraceback (most recent call last):  File "/var/task/lambda_function.py", line 21, in lambda_handler    alarm_name = message['AlarmName']
I tried giving plain json
{
"AlarmName": "PublishedAlarm",
"NewStateValue": "OK",
"NewStateReason": "This alarm is published"
}
I tried giving a stringified JSON
"{\"AlarmName\": \"PublishedAlarm\",\"NewStateValue\": \"OK\",\"NewStateReason\": \"This alarm is published\"}"
I tried choosing Custom payload for each delivery message structure and then gave the following message body
{
"default": "Sample fallback message",
"email": "Sample message for email endpoints",
"sqs": "Sample message for Amazon SQS endpoints",
"lambda": "{\"AlarmName\": \"PublishedAlarm\",\"NewStateValue\": \"OK\",\"NewStateReason\": \"This alarm is published\"}",
"http": "Sample message for HTTP endpoints",
"https": "Sample message for HTTPS endpoints",
"sms": "Sample message for SMS endpoints",
"firehose": "Sample message for Amazon Kinesis Data Firehose endpoints",
"APNS": "{\"aps\":{\"alert\": \"Sample message for iOS endpoints\"} }",
"APNS_SANDBOX": "{\"aps\":{\"alert\":\"Sample message for iOS development endpoints\"}}",
"APNS_VOIP": "{\"aps\":{\"alert\":\"Sample message for Apple VoIP endpoints\"}}",
"APNS_VOIP_SANDBOX": "{\"aps\":{\"alert\": \"Sample message for Apple VoIP development endpoints\"} }",
"MACOS": "{\"aps\":{\"alert\":\"Sample message for MacOS endpoints\"}}",
"MACOS_SANDBOX": "{\"aps\":{\"alert\": \"Sample message for MacOS development endpoints\"} }",
"GCM": "{ \"data\": { \"message\": \"Sample message for Android endpoints\" } }",
"ADM": "{ \"data\": { \"message\": \"Sample message for FireOS endpoints\" } }",
"BAIDU": "{\"title\":\"Sample message title\",\"description\":\"Sample message for Baidu endpoints\"}",
"MPNS": "<?xml version=\"1.0\" encoding=\"utf-8\"?><wp:Notification xmlns:wp=\"WPNotification\"><wp:Tile><wp:Count>ENTER COUNT</wp:Count><wp:Title>Sample message for Windows Phone 7+ endpoints</wp:Title></wp:Tile></wp:Notification>",
"WNS": "<badge version=\"1\" value=\"42\"/>"
}
Nothing worked. I've also subscribed an email address to the topic and I'm getting emails without any issues.
How can I simulate the test event given in lambda event templates from the SNS?
When you send your plain json message using SNS, it will be delivered to lambda in in the format:
'Message': '{\n "AlarmName": "PublishedAlarm",\n "NewStateValue": "OK",\n "NewStateReason": "This alarm is published"\n}'
You can parse it using ast' literal_eval method:
import ast
#...
#...
def lambda_handler(event, context):
logger.info("Event: " + str(event))
message = event['Records'][0]['Sns']['Message']
logger.info("Message: " + str(message))
message = ast.literal_eval(event['Records'][0]['Sns']['Message'])
alarm_name = message['AlarmName']
new_state = message['NewStateValue']
reason = message['NewStateReason']
#...
#...

how to parse more complex human-oriented text output to machine-friently style?

This is the question about how to parse "unparseable" output into json, or to something easily consumable as json. This is "little" bit behind trivial stuff, so I'd like to know, how do you solve these things in principle, it's not about this specific example only. But example:
We have this command, which shows data about audio inputs:
pacmd list-sink-inputs
it prints something like this:
2 sink input(s) available.
index: 144
driver: <protocol-native.c>
flags:
state: RUNNING
sink: 4 <alsa_output.pci-0000_05_00.0.analog-stereo>
volume: front-left: 15728 / 24% / -37.19 dB, front-right: 15728 / 24% / -37.19 dB
balance 0.00
muted: no
current latency: 70.48 ms
requested latency: 210.00 ms
sample spec: float32le 2ch 44100Hz
channel map: front-left,front-right
Stereo
resample method: copy
module: 13
client: 245 <MPlayer>
properties:
media.name = "UNREAL! Tetris Theme on Violin and Guitar-TnDIRr9C83w.webm"
application.name = "MPlayer"
native-protocol.peer = "UNIX socket client"
native-protocol.version = "32"
application.process.id = "1543"
application.process.user = "mmucha"
application.process.host = "vbDesktop"
application.process.binary = "mplayer"
application.language = "C"
window.x11.display = ":0"
application.process.machine_id = "720184179caa46f0a3ce25156642f7a0"
application.process.session_id = "2"
module-stream-restore.id = "sink-input-by-application-name:MPlayer"
index: 145
driver: <protocol-native.c>
flags:
state: RUNNING
sink: 4 <alsa_output.pci-0000_05_00.0.analog-stereo>
volume: front-left: 24903 / 38% / -25.21 dB, front-right: 24903 / 38% / -25.21 dB
balance 0.00
muted: no
current latency: 70.50 ms
requested latency: 210.00 ms
sample spec: float32le 2ch 48000Hz
channel map: front-left,front-right
Stereo
resample method: speex-float-1
module: 13
client: 251 <MPlayer>
properties:
media.name = "Trombone Shorty At Age 13 - 2nd Line-k9YUi3UhEPQ.webm"
application.name = "MPlayer"
native-protocol.peer = "UNIX socket client"
native-protocol.version = "32"
application.process.id = "2831"
application.process.user = "mmucha"
application.process.host = "vbDesktop"
application.process.binary = "mplayer"
application.language = "C"
window.x11.display = ":0"
application.process.machine_id = "720184179caa46f0a3ce25156642f7a0"
application.process.session_id = "2"
module-stream-restore.id = "sink-input-by-application-name:MPlayer"
very nice. But we don't want to show user all of this, we just want to show index (id of input), application.process.id, application.name and media.name, in some reasonable format. It would be great to parse it somehow to json, but even if I preprocess it somehow, the jq is way beyond my capabilities and quite complex. I tried multiple approaches using jq, with regex or without, but I wasn't able to finish it. And I guess we cannot rely on order or presence of all fields.
I was able to get the work "done", but it's messy, inefficient, and namely expects no semicolons in media name or app name. Not acceptable solution, but this is the only thing I was able to bring to the "end".
incorrect solution:
cat exampleOf2Inputs |
grep -e "index: \|application.process.id = \|application.name = \|media.name = " |
sed "s/^[ \t]*//;s/^\([^=]*\) = /\1: /" |
tr "\n" ";" |
sed "s/$/\n/;s/index:/\nindex:/g" |
tail -n +2 |
while read A; do
index=$(echo $A|sed "s/^index: \([0-9]*\).*/\1/");
pid=$(echo $A|sed 's/^.*application\.process\.id: \"\([0-9]*\)\".*$/\1/');
appname=$(echo $A|sed 's/^.*application\.name: \"\([^;]*\)\".*$/\1/');
medianame=$(echo $A|sed 's/^.*media\.name: \"\([^;]*\)\".*$/\"\1\"/');
echo "pid=$pid index=$index appname=$appname medianame=$medianame";
done
~ I grepped the interessant part, replaced newlines with semicolon, split to multiple lines, and just extract the data multiple times using sed. Crazy.
Here the output is:
pid=1543 index=144 appname=MPlayer medianame="UNREAL! Tetris Theme on Violin and Guitar-TnDIRr9C83w.webm"
pid=2831 index=145 appname=MPlayer medianame="Trombone Shorty At Age 13 - 2nd Line-k9YUi3UhEPQ.webm"
which is easily convertable to any format, but the question was about json, so to:
[
{
"pid": 1543,
"index": 144,
"appname": "MPlayer",
"medianame": "UNREAL! Tetris Theme on Violin and Guitar-TnDIRr9C83w.webm"
},
{
"pid": 2831,
"index": 145,
"appname": "MPlayer",
"medianame": "Trombone Shorty At Age 13 - 2nd Line-k9YUi3UhEPQ.webm"
}
]
Now I'd like to see, please, how are these things done correctly.
If the input is as reasonable as shown in the Q, the following approach that only uses jq should be possible.
An invocation along the following lines is assumed:
jq -nR -f parse.jq input.txt
def parse:
def interpret:
if . == null then .
elif startswith("\"") and endswith("\"")
then .[1:-1]
else tonumber? // .
end;
(capture( "(?<key>[^\t:= ]*)(: | = )(?<value>.*)" ) // null)
| if . then .value = (.value | interpret) else . end
;
# Construct one object for each "segment"
def construct(s):
[ foreach (s, 0) as $kv (null;
if $kv == 0 or $kv.index
then .complete = .accumulator | .accumulator = $kv
else .complete = null | .accumulator += $kv
end;
.complete // empty ) ]
;
construct(inputs | parse | select(.) | {(.key):.value})
| map( {pid: .["application.process.id"],
index,
appname: .["application.name"],
medianame: .["media.name"]} )
With the example input, the output would be:
[
{
"pid": "1543",
"index": 144,
"appname": "MPlayer",
"medianame": "UNREAL! Tetris Theme on Violin and Guitar-TnDIRr9C83w.webm"
},
{
"pid": "2831",
"index": 145,
"appname": "MPlayer",
"medianame": "Trombone Shorty At Age 13 - 2nd Line-k9YUi3UhEPQ.webm"
}
]
Brief explanation
parse parses one line. It assumes that whitespace (blank and tab characters) on each line before the key name can be ignored.
construct is responsible for grouping the lines (presented as a stream of key-value single-key objects) corresponding to a particular value of “index”. It produces an array of objects, one for each value of “index”.
I don't know about "correctly", but this is what I'd do:
pacmd list-sink-inputs | awk '
BEGIN { print "[" }
function print_record() {
if (count++) {
print " {"
printf " %s,\n", print_number("pid")
printf " %s,\n", print_number("index")
printf " %s,\n", print_string("appname")
printf " %s\n", print_string("medianame")
print " },"
}
delete record
}
function print_number(key) { return sprintf("\"%s\": %d", key, record[key]) }
function print_string(key) { return sprintf("\"%s\": \"%s\"", key, record[key]) }
function get_quoted_value() {
if (match($0, /[^"]+"$/))
return substr($0, RSTART, RLENGTH-1)
else
return "?"
}
$1 == "index:" { print_record(); record["index"] = $2 }
$1 == "application.process.id" { record["pid"] = get_quoted_value() }
$1 == "application.name" { record["appname"] = get_quoted_value() }
$1 == "media.name" { record["medianame"] = get_quoted_value() }
END { print_record(); print "]" }
' |
tac | awk '/},$/ && !seen++ {sub(/,$/,"")} 1' | tac
where the tac|awk|tac line removes the trailing comma from the last JSON object in the list.
[
{
"pid": 1543,
"index": 144,
"appname": "MPlayer",
"medianame": "UNREAL! Tetris Theme on Violin and Guitar-TnDIRr9C83w.webm"
},
{
"pid": 2831,
"index": 145,
"appname": "MPlayer",
"medianame": "Trombone Shorty At Age 13 - 2nd Line-k9YUi3UhEPQ.webm"
}
]
You could just pipe your output into:
sed -E '
s/pid=([0-9]+) index=([0-9]+) appname=([^ ]+) medianame=(.*)/{"pid": \1, "index": \2, "appname": "\3", "medianame": \4},/
1s/^/[/
$s/,$/]/
' | jq .

Keep track of the last read record in MySQL

I have 2 databases in MySQL:
1) An input Latitude-Longitude_dB ('latlong_db', henceforth): It has the latitude and longitude of each reading from a GPS tracking device.
2) A Weather_db: I read the input latlongs from dB1, and calculate 'current' weather data for each pair of latlongs (eg: humidity, cloud_coverage) . This weather data is written into a Weather_db.
The issue is: I need to keep track of which record (which 'input latlong') was read last. This is so that I don't recalculate weather_data for the latlongs that I've already covered. How do I keep track of the last read input_latlong?
Thank you so much.
Edit:
1) For those who have been asking about the 'database v/s table' question, the answer is that I am reading from 1 database and writing into the 2nd database. The 'config.json' to connect to the 2 databases is as follows:
{
"Tracker_ds_locallatlongdb": {
"database": "ds_testdb1",
"host": "XXXXXXXXXXX",
"port": XXXX,
"user": "XXXX",
"password": "XXXXX"
},
"Tracker_ds_localweatherdb": {
"database": "ds_testdb2",
"host": "XXXXXXX",
"port": XXXX,
"user": "XXXX",
"password": "XXXXX"
}
}
2) My Python script to read from the input_latlong_db and write into the weather_db is outlined as follows. I am using OpenWeatherMap API to calculate weather data for given latitudes and longitudes:
from pyowm import OWM
import json
import time
import pprint
import pandas as pd
import mysql.connector
from mysql.connector import Error
api_key = 'your api key'
def get_weather_data(my_lat, my_long):
owm= OWM(api_key)
obs= owm.weather_at_coords(my_lat.item() , my_long.item() ) #Use: <numpy.ndarray>.item:
w= obs.get_weather()
l= obs.get_location()
city= l.get_name()
cloud_coverage =w.get_clouds()
.
.
.
w_datatoinsert= [my_lat, my_long, w_latitude, w_longitude, city, weather_time_gmt,call_time_torontotime,
short_status, detailed_status,
temp_celsius, cloud_coverage, humidity, wind_deg, wind_speed,
snow, rain, atm_pressure, sea_level_pressure,sunset_time_gmt ] #15 + act_latitude + act_longitude
return w_datatoinsert
# ------------------------------------------------------------------------------------------------------------------------------------
spec_creds_1= {}
spec_creds_2= {}
def operation():
with open('C:/Users/config.json') as config_file:
creds_dict= json.load(config_file)
spec_creds_1= creds_dict['Tracker_ds_locallatlongdb']
spec_creds_2= creds_dict['Tracker_ds_localweatherdb']
try:
my_conn_1= mysql.connector.connect(**spec_creds_1 )
if (my_conn_1.is_connected()):
info_1= my_conn_1.get_server_info()
print("Connected ..now reading the local input_latlong_db: ", info_1)
try:
my_conn_2= mysql.connector.connect(**spec_creds_2)
if (my_conn_2.is_connected()):
info_2= my_conn_2.get_server_info()
print('Connected to write into the local weather_db: ', info_2)
cursor_2= my_conn_2.cursor()
readings_df= pd.read_sql("SELECT latitude, longitude FROM readings_table_19cols;", con= my_conn_1)
for index, row in readings_df.iterrows():
gwd= get_weather_data(row['latitude'], row['longitude'])
q= "INSERT INTO weather_table_19cols VALUES(" + ",".join(["%s"]*len(gwd)) + " ); "
cursor_2.execute(q, gwd)
my_conn_2.commit()
except Error as e:
print("Error while connecting to write into the local weather_db: ", e)
finally:
if (my_conn_2.is_connected()):
cursor_2.close()
my_conn_2.close()
print("Wrote 1 record to the local weather_db.")
except Error as e:
print("Error connecting to the local input latlong_db: ", e)
finally:
if (my_conn_1.is_connected()):
my_conn_1.close() # no cursor present for 'my_conn_1'
print("Finished reading all the input latlongs ...and finished inserting ALL the weather data.")
#-------------------------------------------------------------------------------
if __name__=="__main__":
operation()
In the Input_latlong_table- the readings_table_19cols: I created an Autoincremented readings_id as the Primary key , and a column called read_flag whose default value was 0.
In the weather_table_19cols: I created an Autoincremented weather_id as the Primary key.
Since my method involves reading an input latlong record and correspondingly writing its weather data into the weather_table, I compared the index of the readings_table_19cols and the weather_table_19cols. If they matched, that means that the input record was read, and I would set the read_flag to 1.
..
for index_1, row_1 in readings_df_1.iterrows():
gwd= get_weather_data(row_1['imei'], row_1['reading_id'] ,row_1['send_time'],row_1['latitude'], row_1['longitude'])
q_2= "INSERT INTO weather_table_23cols(my_imei, my_reading_id, actual_latitude, actual_longitude, w_latitude, w_longitude, city, weather_time_gmt, OBD_send_time_gmt, call_time_torontotime, \
short_status, detailed_status, \
temp_celsius, cloud_coverage, humidity, wind_deg, wind_speed, \
snow, rain, atm_pressure, sea_level_pressure,sunset_time_gmt) VALUES(" + ",".join(["%s"]*len(gwd)) + " ); "
q_1b= "UPDATE ds_testdb1.readings_table_22cols re, ds_testdb2.weather_table_23cols we \
SET re.read_flag=1 WHERE (re.reading_id= we.weather_id);" # use the prefix 'db_name.table_name' if 1 cursor is being used for 2 different db's
cursor_2.execute(q_2, gwd)
my_conn_2.commit()
cursor_1.execute(q_1b) # use Cursor_1 for 1st query
my_conn_1.commit()

Livy Server: return a dataframe as JSON?

I am executing a statement in Livy Server using HTTP POST call to localhost:8998/sessions/0/statements, with the following body
{
"code": "spark.sql(\"select * from test_table limit 10\")"
}
I would like an answer in the following format
(...)
"data": {
"application/json": "[
{"id": "123", "init_date": 1481649345, ...},
{"id": "133", "init_date": 1481649333, ...},
{"id": "155", "init_date": 1481642153, ...},
]"
}
(...)
but what I'm getting is
(...)
"data": {
"text/plain": "res0: org.apache.spark.sql.DataFrame = [id: string, init_date: timestamp ... 64 more fields]"
}
(...)
Which is the toString() version of the dataframe.
Is there some way to return a dataframe as JSON using the Livy Server?
EDIT
Found a JIRA issue that addresses the problem: https://issues.cloudera.org/browse/LIVY-72
By the comments one can say that Livy does not and will not support such feature?
I recommend using the built-in (albeit hard to find documentation for) magics %json and %table:
%json
session_url = host + "/sessions/1"
statements_url = session_url + '/statements'
data = {
'code': textwrap.dedent("""\
val d = spark.sql("SELECT COUNT(DISTINCT food_item) FROM food_item_tbl")
val e = d.collect
%json e
""")}
r = requests.post(statements_url, data=json.dumps(data), headers=headers)
print r.json()
%table
session_url = host + "/sessions/21"
statements_url = session_url + '/statements'
data = {
'code': textwrap.dedent("""\
val x = List((1, "a", 0.12), (3, "b", 0.63))
%table x
""")}
r = requests.post(statements_url, data=json.dumps(data), headers=headers)
print r.json()
Related: Apache Livy: query Spark SQL via REST: possible?
I don't have a lot of experience with Livy, but as far as I know this endpoint is used as an interactive shell and the output will be a string with the actual result that would be shown by a shell. So, with that in mind, I can think of a way to emulate the result you want, but It may not be the best way to do it:
{
"code": "println(spark.sql(\"select * from test_table limit 10\").toJSON.collect.mkString(\"[\", \",\", \"]\"))"
}
Then, you will have a JSON wrapped in a string, so your client could parse it.
I think in general your best bet is to write your output to a database of some kind. If you write to a randomly named table, you could have your code read it after the script is done.