Folks,
The following python script is terminating with
job state = FAILED
and
Last State Change: Access denied checking streaming input path: s3n://elasticmapreduce/samples/wordcount/input/
Code:
import boto
import boto.emr
from boto.emr.step import StreamingStep
from boto.emr.bootstrap_action import BootstrapAction
import time
S3_BUCKET="mytesetbucket123asdf"
conn = boto.connect_emr()
step = StreamingStep(
name='Wordcount',
mapper='s3n://elasticmapreduce/samples/wordcount/wordSplitter.py',
reducer='aggregate',
input='s3n://elasticmapreduce/samples/wordcount/input/',
output='s3n://' + S3_BUCKET + '/wordcount/output/2013-10-25')
jobid = conn.run_jobflow(
name="test",
log_uri="s3://" + S3_BUCKET + "/logs/",
visible_to_all_users="True",
steps = [step],)
state = conn.describe_jobflow(jobid).state
print "job state = ", state
print "job id = ", jobid
while state != u'COMPLETED':
print time.localtime()
time.sleep(10)
state = conn.describe_jobflow(jobid).state
print conn.describe_jobflow(jobid)
print "job state = ", state
print "job id = ", jobid
print "final output can be found in s3://" + S3_BUCKET + "/output" + TIMESTAMP
print "try: $ s3cmd sync s3://" + S3_BUCKET + "/output" + TIMESTAMP + " ."
The problem is somewhere in boto... If we specify IAM user instead of using Roles, job works perfectly. EMR supports IAM Roles ofcourse... and the IAM role we tested with has full rights to execute any task, so its not a mis-configuration issue...
Related
I am trying to set up an automated alert emailing system for a project and I can format the email properly when it's plain text but when I try to add variables it won't format properly. I am pulling data from a JSON file and I can pull the data I just can't combine it with the email inside the triple quotes (""").
Here is the plain text
message = """\
Subject: AMLD Alert
Something is wrong."""
But when I add in my variables it will not fill in the body, but the subject will.
message = 'Subject: AMLD Alert for ' + d[0]['Project Name'] + ' Project'
'Car: ' + d[0]['CarID'] +' Driven by: ' + d[0]['DriverID'] + ' is sending alert '+ d[0]['Message']
Here is the entire script if it will help.
import json
import smtplib
import ssl
import socket
socket.getaddrinfo('localhost', 8080)
#Webiste use to help me build script
#https://realpython.com/python-send-email/
port = 587 # For starttls
smtp_server = "smtp.outlook.com"
sender_email = "MyWorkOutlookEmail" # Enter your address
password = 'Password'
receiver_email = "MyPersonalGmail" # Enter receiver address
context = ssl.create_default_context()
#Start of ITTT Code
with open('C:/Python/Messaging/Mes_V1.json') as f:
d = json.load(f)
if d[0]['Alert'] == "High":
print('Sending high alert email...\n')
message = 'Subject: AMLD Alert for ' + d[0]['Project Name'] + ' Project'
'Car: ' + d[0]['CarID'] +' Driven by: ' + d[0]['DriverID'] + ' is sending alert '+ d[0]['Message']
elif d[0]['Alert'] == "Medium":
print('Sending medium alert email...\n')
else:
print('Sending low alert email...\n')
context = ssl.create_default_context()
with smtplib.SMTP(smtp_server, port) as server:
print('Connecting to Server...\n')
server.starttls(context=context)
print('Logging in...\n')
server.login(sender_email, password)
server.sendmail(sender_email, receiver_email, message)
print('Email sent.\n')
I'm not sure if Outlook is affecting it in any way or if I need to format the message differently, any help would be very appreciated!
On tutorial 01 from flow:Tutorial 01.
I executed the code
flow_params = dict(
exp_tag='ring_example',
env_name=AccelEnv,
network=RingNetwork,
simulator='traci',
sim=sim_params,
env=env_params,
net=net_params,
veh=vehicles,
initial=initial_config,
tls=traffic_lights,
)
# number of time steps
flow_params['env'].horizon = 3000
exp = Experiment(flow_params)
# run the sumo simulation
_ = exp.run(1, convert_to_csv=True)
I got an error afterward, here is the error
Error during start: [Errno 2] No such file or directory: '.../kernel/network/debug/cfg/ring_example_20201208-1332481607405568.58399.net.xml' Retrying in 1 seconds...
How should it be generated or where can it be found?
This is an issue with my file naming conversion. Apparently, the command to be called in
subprocess.call(
[
'netconvert -c "' + self.net_path + self.cfgfn +
'" --output-file="' + self.cfg_path + self.netfn +
'" --no-internal-links="false"'
],
stdout=subprocess.DEVNULL,
shell=True)
requires no spacing. In my case, I have my folder named "Machine Learning."
Suppose the following scenario in using Zabbix 4.2. We have a core switch, two distributed switches and 20 access switches, where the distributed switches are connected to the core and 10 access switches are connected to each distributed switch. I am monitoring all of them using SNMP v2c and using the template cisco switches (the official one). Now the problem arises as I cannot define device dependency in zabbix easily. By easily, I mean that if a distributed switch goes out, I want to have the alarm for that device and not for all access switches connected to it. I could define it as follows. Change the triggers for each device and made them dependent on the corresponding trigger for distributed switches. However, this is too time consuming. What should I do? Any help is appreciated.
You are right, there isn't an easy way to set this kind of dependancy.
I had to manage the same situation a while ago and I wrote a python dependancy setter which uses a "dependent hostgroup <--> master host" logic.
You can modify it to fit your needs (see masterTargetTriggerDescription and slaveTargetTriggerDescription for the dependancy targets), it works but contains little error checking: use at your own risk!
import csv
import re
import json
from zabbix.api import ZabbixAPI
# Zabbix Server endpoint
zabbixServer = 'https://yourzabbix/zabbix/'
zabbixUser = 'admin'
zabbixPass = 'zabbix'
zapi = ZabbixAPI(url=zabbixServer, user=zabbixUser, password=zabbixPass)
# Hostgrop variables - to reference IDs while building API parameters
hostGroupNames = [] # list = array
hostGroupId = {} # dict = associative array
# Csv file for dep settings - see the format:
"""
Hostgroup;Master
ACCESS_1;DistSwitch1
ACCESS_2;DistSwitch1
ACCESS_5;DistSwitch2
ACCESS_6;DistSwitch2
DIST;CoreSwitch1
"""
fileName = 'dependancy.csv'
masterTargetTriggerDescription = '{HOST.NAME} is unavailable by ICMP'
slaveTargetTriggerDescription = '{HOST.NAME} is unavailable by ICMP|Zabbix agent on {HOST.NAME} is unreachable'
# Read CSV file
hostFile = open(fileName)
hostReader = csv.reader(hostFile, delimiter=';', quotechar='|')
hostData = list(hostReader)
# CSV Parsing
for line in hostData:
hostgroupName = line[0]
masterName = line[1]
slaveIds = []
masterId = zapi.get_id('host', item=masterName, with_id=False, hostid=None)
hostGroupId = zapi.get_id('hostgroup', item=hostgroupName, with_id=False, hostid=None)
masterTriggerObj = zapi.trigger.get(hostids=masterId, filter=({'description': masterTargetTriggerDescription}) )
print "Group: " + hostgroupName + " - ID: " + str(hostGroupId)
print "Master host: " + masterName + " - ID: " + str(masterId)
print "Master trigger: " + masterTriggerObj[0]['description'] + " - ID: " + str(masterTriggerObj[0]['triggerid'])
# cycle through slave hosts
hostGroupObj = zapi.hostgroup.get(groupids=hostGroupId, selectHosts='extend')
for host in hostGroupObj[0]['hosts']:
#exclude master
if host['hostid'] != str(masterId):
print " - Host Name: " + host['name'] + " - ID: " + host['hostid'] + " - MASTER: " + str(masterId)
# cycle for all slave's triggers
slaveTargetTriggerObj = zapi.trigger.get(hostids=host['hostid'])
#print json.dumps(slaveTargetTriggerObj)
for slaveTargetTrigger in slaveTargetTriggerObj:
# search for dependancy targets
if re.search(slaveTargetTriggerDescription, slaveTargetTrigger['description'] ,re.IGNORECASE):
print " - Trigger: " + slaveTargetTrigger['description'] + " - ID: " + slaveTargetTrigger['triggerid']
# Clear existing dep. from the trigger, then create the new dep.
clear = zapi.trigger.deletedependencies(triggerid=slaveTargetTrigger['triggerid'].encode())
result = zapi.trigger.adddependencies(triggerid=slaveTargetTrigger['triggerid'].encode(), dependsOnTriggerid=masterTriggerObj[0]['triggerid'])
print "----------------------------------------"
print ""
I updated the code contributed by Simone Zabberoni and rewritten it to work with Python 3, PyZabbix, and YAML.
#!/usr/bin/python3
import re
import yaml
#https://pypi.org/project/py-zabbix/
from pyzabbix import ZabbixAPI
# Zabbix Server endpoint
zabbix_server = 'https://zabbix.example.com/zabbix/'
zabbix_user = 'zbxuser'
zabbix_pass = 'zbxpassword'
# Create ZabbixAPI class instance
zapi = ZabbixAPI(zabbix_server)
# Enable HTTP auth
zapi.session.auth = (zabbix_user, zabbix_pass)
# Login (in case of HTTP Auth, only the username is needed, the password, if passed, will be ignored)
zapi.login(zabbix_user, zabbix_pass)
# Hostgrop variables - to reference IDs while building API parameters
hostGroupNames = [] # list = array
hostGroupId = {} # dict = associative array
# yaml file for dep settings - see the format:
"""
pvebar16 CTs:
master: pvebar16.example.com
masterTargetTriggerDescription: 'is unavailable by ICMP'
slaveTargetTriggerDescription: 'is unavailable by ICMP|Zabbix agent is unreachable for 5 minutes'
"""
fileName = 'dependancy.yml'
with open('dependancy.yml') as f:
hostData = yaml.load(f)
for groupyml in hostData.keys():
masterTargetTriggerDescription = hostData[groupyml]['masterTargetTriggerDescription']
slaveTargetTriggerDescription = hostData[groupyml]['slaveTargetTriggerDescription']
masterName = hostData[groupyml]['master']
hostgroupName = groupyml
slaveIds = []
masterId = zapi.host.get(filter={'host': masterName},output=['hostid'])[0]['hostid']
hostGroupId = zapi.hostgroup.get(filter={'name': hostgroupName},output=['groupid'])[0]['groupid']
masterTriggerObj = zapi.trigger.get(host=masterName, filter={'description': masterTargetTriggerDescription}, output=['triggerid','description'])
print("Group: " + hostgroupName + " - ID: " + str(hostGroupId))
print("Master host: " + masterName + " - ID: " + str(masterId))
print("Master trigger: " + masterTriggerObj[0]['description'] + " - ID: " + str(masterTriggerObj[0]['triggerid']))
# cycle through slave hosts
hostGroupObj = zapi.hostgroup.get(groupids=hostGroupId, selectHosts='extend')
for host in hostGroupObj[0]['hosts']:
#exclude master
if host['hostid'] != str(masterId):
print(" - Host Name: " + host['name'] + " - ID: " + host['hostid'] + " - MASTER: " + str(masterId))
# cycle for all slave's triggers
slaveTargetTriggerObj = zapi.trigger.get(hostids=host['hostid'])
for slaveTargetTrigger in slaveTargetTriggerObj:
# search for dependancy targets
if re.search(slaveTargetTriggerDescription, slaveTargetTrigger['description'] ,re.IGNORECASE):
print(" - Trigger: " + slaveTargetTrigger['description'] + " - ID: " + slaveTargetTrigger['triggerid'])
# Clear existing dep. from the trigger, then create the new dep.
clear = zapi.trigger.deletedependencies(triggerid=slaveTargetTrigger['triggerid'])
result = zapi.trigger.adddependencies(triggerid=slaveTargetTrigger['triggerid'], dependsOnTriggerid=masterTriggerObj[0]['triggerid'])
print("----------------------------------------")
print("")
I have local json file, I am beginner stage at python and I need to develop a code for while running python code it continuously checks the json file and if any changes in Json file I need to show the changes in Python console.
Eg.,
{
"a":2,
"b":3
}
if I change "a":3
python output:
change detected at key a and value 3.
I am missing logic here. and Thank you in advance
Hia, Seems like you've got a project going on. Depending on what values you want to change and how deep in the JSON they are could add more work to the task. Let me see what i can come up with...
import essentials # pip install mknxgn_essentials (not gonna lie, i made this module. biased...)
import time
import os
recordedjson = essentials.EsFileObject("Tools/Json.json").json
recoredchanges = essentials.EsFileObject("Tools/Changes.json")
if recoredchanges.json == False: # Used to save changes, also makes sure to keep previous logs
recoredchanges.setjson([])
onchange_record_new_json = True # makes the new json the one you want to compare after each change
while True:
newjson = essentials.EsFileObject('Tools/Json.json').json
if recordedjson != newjson:
changes = 0
keychanges = 0
newkeys = 0
removedkeys = 0
valuechanges = 0
print("File has been changed! - Collecting Changes")
for obj in recordedjson:
if obj not in newjson:
print("Key Removed From New Json Key:", obj)
keychanges += 1
removedkeys += 1
changes += 1
else:
if recordedjson[obj] != newjson[obj]:
print("Value Change!")
print("Previous- Key: ", obj)
print("# Value:", recordedjson[obj])
print(" New- Value:", newjson[obj])
valuechanges += 1
changes += 1
for obj in newjson:
if obj not in recordedjson:
if obj not in newjson:
print("New Key Introduced- Key:", obj)
keychanges += 1
removedkeys += 1
changes += 1
changelog = "Change Count: " + str(changes)
changelog += " Key Changes: " + str(keychanges)
changelog += " New Keys: " + str(newkeys)
changelog += " Removed Keys: " + str(removedkeys)
changelog += " Value Changes: " + str(valuechanges)
print(changelog)
record = {"Change Time": essentials.EsTimeObject().string}
record["User Readable Time Change"] = essentials.EsTimeObject().readable
record["Change Log"] = changelog
recoredchanges.json.append(record)
recoredchanges.save()
time.sleep(10)
if onchange_record_new_json:
recordedjson = newjson
print("Waiting For Change")
time.sleep(1)
os.system("cls") # cls for windows, clear for linux and so on...
man is it annoying to post code on this site..
There was an earlier question on this, but the asker was just overwriting their output and solved their own problem.
I'm using a subprocess.popen to read video information and write the output to a json. It works fine on MOST videos, but on others is returning an empty string on others - even though it runs fine from the command line. I tried it several times and am getting the data fine through the command line.
Here's the relevant part of the script:
out_prj.write('[')
for m, i in enumerate(files):
print i
out_prj.write('{"$type":"BatchProcessor.Job, BatchProcessor","Id":0,"Ver":1.02,"CurrentTask":0,"IsSelected":true,"TaskList":[')
f_name = os.path.basename(i[0])
f_json = out_folder + os.sep + "06_Output" + os.sep + os.path.basename(i[0]).split(".")[0] + ".json"
trans_f = out_folder + os.sep + "04_Video" + os.sep + os.path.basename(i[0]).split(".")[0] + "-tr.ts"
trans_f_out = out_folder + os.sep + "06_Output" + os.sep + os.path.basename(i[0]).split(".")[0] + "-tr-out.ts"
ffprobe = 'ffprobe.exe'
command = [ffprobe, '-v', 'quiet', '-print_format', 'json', '-show_format', '-show_streams', i[0]]
p = sp.Popen(command, stdout=sp.PIPE, stderr=sp.PIPE, shell=True)
out, err = p.communicate()
io = cStringIO.StringIO(out)
info = json.load(io)
print info
filea = open(f_json, 'w')
filea.write(json.dumps(info))
filea.close()
f = open(f_json)
b = json.load(f)
print b
#########################
###################
f_format = str(b['streams'][0]['codec_long_name'])
Your code ignores error messages (err variable). print err or don't redirect stderr to see them.
Unrelated: the json handling in your code is insane: most operations are redundant.
To save output of the subprocess to a file:
import os
from subprocess import check_call
f_json = os.path.join(out_folder, "06_Output",
os.path.splitext(f_name)[0] + ".json")
with open(f_json, 'wb', 0) as file:
check_call(command, stdout=file)
Note: shell=True is not necessary here. If subprocess can't find ffprobe.exe then specify the full path e.g. (use the path appropriate for your system):
ffprobe = r'C:\Program Files\Real\RealPlayer\RPDS\Tools\ffmpeg\ffprobe.exe'
Note: r'' -- a raw string literal is used to avoid doubling the backslashes.