Iterating through describe_instances() to print key & value boto3 - json

I am currently working on a python script to print pieces of information on running EC2 instances on AWS using Boto3. I am trying to print the InstanceID, InstanceType, and PublicIp. I looked through Boto3's documentation and example scripts so this is what I am using:
import boto3
ec2client = boto3.client('ec2')
response = ec2client.describe_instances()
for reservation in response["Reservations"]:
for instance in reservation["Instances"]:
instance_id = instance["InstanceId"]
instance_type = instance["InstanceType"]
instance_ip = instance["NetworkInterfaces"][0]["Association"]
print(instance)
print(instance_id)
print(instance_type)
print(instance_ip)
When I run this, "instance" prints one large block of json code, my instanceID, and type. But I am getting an error since adding NetworkInterfaces.
instance_ip = instance["NetworkInterfaces"][0]["Association"]
returns:
Traceback (most recent call last):
File "/Users/me/AWS/describeInstances.py", line 12, in <module>
instance_ip = instance["NetworkInterfaces"][0]["Association"]
KeyError: 'Association'
What am I doing wrong while trying to print the PublicIp?
Here is the structure of NetworkInterfaces for reference:
The full Response Syntax for reference can be found here (https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Client.describe_instances)

Association man not always may be present. Also an instance may have more then one interface. So your working loop could be:
for reservation in response["Reservations"]:
for instance in reservation["Instances"]:
instance_id = instance["InstanceId"]
instance_type = instance["InstanceType"]
#print(instance)
print(instance_id, instance_type)
for network_interface in instance["NetworkInterfaces"]:
instance_ip = network_interface.get("Association", "no-association")
print(' -', instance_ip)

Related

Why does Pycryptodome MAC check fail when encrypting and decrypting JSON files?

I am trying to do encrypt some JSON data with AES-256, using a password hashed with pbkdf2_sha256 as the key. I want to store the data in a file, be able to load it up, decrypt it, alter it, encrypt it, store it, and repeat.
I am using the passlib and pycryptodome libraries with python 3.8. The following test occurs inside a docker container and throws an error I haven't been able to correct
Does anyone have any clues on how I can improve my code (and knowledge)?
Test.py:
import os, json
from Crypto.PublicKey import RSA
from Crypto.Cipher import AES
from passlib.hash import pbkdf2_sha256
def setJsonData(jsonData, jsonFileName):
with open(jsonFileName, 'wb') as jsonFile:
password = 'd'
key = pbkdf2_sha256.hash(password)[-16:]
data = json.dumps(jsonData).encode("utf8")
cipher = AES.new(key.encode("utf8"), AES.MODE_EAX)
ciphertext, tag = cipher.encrypt_and_digest(data)
[ jsonFile.write(x) for x in (cipher.nonce, tag, ciphertext) ]
def getJsonData(jsonFileName):
with open(jsonFileName, 'rb') as jsonFile:
password = 'd'
key = pbkdf2_sha256.hash(password)[-16:]
nonce, tag, ciphertext = [ jsonFile.read(x) for x in (16, 16, -1) ]
cipher = AES.new(key.encode("utf8"), AES.MODE_EAX, nonce)
data = cipher.decrypt_and_verify(ciphertext, tag)
return json.loads(data)
dictTest = {}
dictTest['test'] = 1
print(str(dictTest))
setJsonData(dictTest, "test")
dictTest = getJsonData("test")
print(str(dictTest))
Output:
{'test': 1}
Traceback (most recent call last):
File "test.py", line 37, in <module>
dictTest = getJsonData("test")
File "test.py", line 24, in getJsonData
data = cipher.decrypt_and_verify(ciphertext, tag)
File "/usr/local/lib/python3.8/site-packages/Crypto/Cipher/_mode_eax.py", line 368, in decrypt_and_verify
self.verify(received_mac_tag)
File "/usr/local/lib/python3.8/site-packages/Crypto/Cipher/_mode_eax.py", line 309, in verify
raise ValueError("MAC check failed")
ValueError: MAC check failed
Research:
Looked into this answer, but I believe my verify() call is in
the right place
I noted that in the python docs, it says:
loads(dumps(x)) != x if x has non-string keys.
but, when I re-run the test with dictTest['test'] = 'a' I have the same error.
I suspected the problem was the json formatting, so I did the same test with a string and didn't make the json.loads and json.dumps calls, but I have the same error
The problem here is that key = pbkdf2_sha256.hash(password)[-16:] hashes the key with a new salt each call. Therefore, the cipher used to encrypt and decrypt the cipher text is going to be different, yielding different data, and thus failing the integrity check.
I changed my key derivation function to the following:
h = SHA3_256.new()
h.update(password.encode("utf-8"))
key = h.digest()

Python3 Error: TypeError: 'str' object is not callable

I'd like to get the latest posts id from a subreddit. Reddit is have basic api for this. You can get json so i want gives data and decode it but i have a error.
root#archi-sunucu:~/yusuf/www# python3 reddit.py
Traceback (most recent call last):
File "reddit.py", line 24, in <module>
json = json.loads(resp.text())
TypeError: 'str' object is not callable
root#archi-sunucu:~/yusuf/www# python3 reddit.py
my code:
url = "https://www.reddit.com/r/" + subreddit + "/" + feed + ".json?sort=" + feed + "&limit=6"
resp = requests.get(url, verify=False)
json = json.loads(resp.text())
print(json["data"]["children"][0]["data"]["id"])
thanks for helps...
You complained that this expression raises an error:
json.loads(resp.text())
Well, let's break that down into something simpler,
so the line number tells us exactly what part of your code is failing.
temp = resp.text()
json.loads(temp)
Now we see that the 2nd line doesn't even execute,
it fails in the 1st line attempting to compute something
to assign to the temporary variable.
Examine resp and its attribute with tools
like help(resp), dir(resp), type(resp.text), repr(resp.text).
You will soon learn the .text attribute is a str.
That is not a callable function, so python raises an error.
Use the value directly, without a call:
json = json.loads(resp.text)

Python Json reference and validation

I'm starting using python to validate some json information, i'm using a json schema with reference but i'm having trouble to reference those files. This is the code :
from os.path import join, dirname
from jsonschema import validate
import jsonref
def assert_valid_schema(data, schema_file):
""" Checks whether the given data matches the schema """
schema = _load_json_schema(schema_file)
return validate(data, schema)
def _load_json_schema(filename):
""" Loads the given schema file """
relative_path = join('schemas', filename).replace("\\", "/")
absolute_path = join(dirname(__file__), relative_path).replace("\\", "/")
base_path = dirname(absolute_path)
base_uri = 'file://{}/'.format(base_path)
with open(absolute_path) as schema_file:
return jsonref.loads(schema_file.read(), base_uri=base_uri, jsonschema=True, )
assert_valid_schema(data, 'grandpa.json')
The json data is :
data = {"id":1,"work":{"id":10,"name":"Miroirs","composer":{"id":100,"name":"Maurice Ravel","functions":["Composer"]}},"recording_artists":[{"id":101,"name":"Alexandre Tharaud","functions":["Piano"]},{"id":102,"name":"Jean-Martial Golaz","functions":["Engineer","Producer"]}]}
And i'm saving the schema and reference file, into a schemas folder :
recording.json :
{"$schema":"http://json-schema.org/draft-04/schema#","title":"Schema for a recording","type":"object","properties":{"id":{"type":"number"},"work":{"type":"object","properties":{"id":{"type":"number"},"name":{"type":"string"},"composer":{"$ref":"artist.json"}}},"recording_artists":{"type":"array","items":{"$ref":"artist.json"}}},"required":["id","work","recording_artists"]}
artist.json :
{"$schema":"http://json-schema.org/draft-04/schema#","title":"Schema for an artist","type":"object","properties":{"id":{"type":"number"},"name":{"type":"string"},"functions":{"type":"array","items":{"type":"string"}}},"required":["id","name","functions"]}
And this is my error :
Connected to pydev debugger (build 181.5281.24)
Traceback (most recent call last):
File "C:\Python\lib\site-packages\proxytypes.py", line 207, in __subject__
return self.cache
File "C:\Python\lib\site-packages\proxytypes.py", line 131, in __getattribute__
return _oga(self, attr)
AttributeError: 'JsonRef' object has no attribute 'cache'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Python\lib\site-packages\jsonref.py", line 163, in callback
base_doc = self.loader(uri)
<MORE>
python version : 3.6.5
windows 7
Ide : intellijIdea
Can somebody help me?
Thank you
I am not sure why, but on Windows, the file:// needs an extra /. So the following change should do the trick
base_uri = 'file:///{}/'.format(base_path)
Arrived at this answer from a solution posted for a related issue in json schema

Converting JSON files to .csv

I've found some data that someone is downloading into a JSON file (I think! - I'm a newb!). The file contains data on nearly 600 football players.
Here you can find the file
In the past, I have downloaded the json file and then used this code:
import csv
import json
json_data = open("file.json")
data = json.load(json_data)
f = csv.writer(open("fix_hists.csv","wb+"))
arr = []
for i in data:
fh = data[i]["fixture_history"]
array = fh["all"]
for j in array:
try:
j.insert(0,str(data[i]["first_name"]))
except:
j.insert(0,'error')
try:
j.insert(1,data[i]["web_name"])
except:
j.insert(1,'error')
try:
f.writerow(j)
except:
f.writerow(['error','error'])
json_data.close()
Sadly, when I do this now in command prompt, i get the following error:
Traceback (most recent call last):
File"fix_hist.py", line 12 (module)
fh = data[i]["fixture_history"]
TypeError: list indices must be integers, not str
Can this be fixed or is there another way I can grab some of the data and convert it to .csv? Specifically the 'Fixture History'? and then 'First'Name', 'type_name' etc.
Thanks in advance for any help :)
Try this tool: http://www.convertcsv.com/json-to-csv.htm
You will need to configure a few things, but should be easy enough.

New SQLAlchemy Add Record Error

Newbie to SQLAlchemy.
I'm having trouble adding a record. I modeled the add after the tutorial which passes multiple values (albeit hard coded values.) Attached is the routine and the error.
StackOverflow thinks my 'explanation to code' ratio is off, so I'm adding additional explanation so I can submit my query.
import pdb
from table import wrl
from sqlalchemy import or_, and_, desc, asc
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
rs = create_engine('credentials', echo=True)
aws = create_engine('credentials', echo=True)
rs_session = sessionmaker(bind=rs)
aws_session = sessionmaker(bind=aws)
rs = rs_session()
aws = aws_session()
# pdb.set_trace()
y = rs.query(wrl).order_by(wrl.UUID_PK).first()
cat = y.Added_Timestamp #now we have the oldest record time stamp value
query_string = cat[:8]+"%" #now we have the oldest record's date i.e. substring(20111215_121212;1;8)
move_me = rs.query(wrl).filter(wrl.Added_Timestamp.like(query_string)).limit(10)
pdb.set_trace()
for x in move_me:
# pdb.set_trace()
wrl_rec = wrl(x.UUID_PK,
x.Web_Request_Headers,
x.Web_Request_Body,
x.Current_Machine,
x.Current_Machine,
x.ResponseBody,
x.Full_Log_Message,
x.Remote_Address,
x.basic_auth_username,
x.Request_Method,
x.Request_URI,
x.Request_Protocol,
x.Time_To_Process_Request,
x.User_ID,
x.Error,
x.Added_Timestamp,
x.Processing_Time_Milliseconds,
x.mysql_timestamp)
aws.add(wrl_rec)
aws.commit()
print 'added %s ' % x.UUID_PK
Traceback (most recent call last):
File "migrate.py", line 47, in <module>
x.mysql_timestamp)
TypeError: __init__() takes exactly 1 argument (19 given)
Any suggestions appreciated.
The problem is not really SA related. My conjecture is that your constructor (wrl.__init__(self, ...)) is either not defined, or does not take any positional arguments, which you are trying to specify when creating this object in wrl_rec.
So basically, the error message is pretty much indicating your problem.
On a side-note, does order_by(wrl.UUID_PK) really return the oldest record by the Timestamp as your comment few lines below indicate? Somehow I highly doubt that.