GPG Public/private key cannot be accessed correctly from AWS Secrets manager via python3 - json

Question

I am using python-gnupg package to create GPG public and private key. The generated private key I am storing in AWS secrets manager as follows.
Key: private_key
value: -----BEGIN PGP PRIVATE KEY BLOCK-----
Version: GnuPG v2.0.22 (GNU/Linux)
lQO+BF37qDIBCADXq0iJVRYFb43+YU8Ts63hDgZl49ZNdnDVhd9H0JMXRHqtPqt9
bbFePPN47NRe6z6GsbPaPmDqEE9l3KjFnSZB/yCii+2wHZR0ij2g3ATbiAbOoQQy
I6bbUADmHtcfIJByoXVoDk489nUPt84Xyp1lHiBfCtUmq4w62Okq6InlRhxjxcEx
VvSXaCY8YnEXUAgNGpvcKHDejGS9V4djh7r7lgJ/Y+3Xb2eepOfiaCx2Cn8ZMI0q
7eWH0MmSeR4ueOLeb79ZKjpJraBfV91XplgHHiM18oECWWwsQCFiwi1GVOLpX6Fh
HIoUyaRAW2vZyFcNnO7iLbetie6fE884lfHxABEBAAH+AwMCO+Qoh7o3GWVga9f2
gHEeuGGH4KB3aspQZt/zwKpx7YlDB/uLd4q7JQt35nIH6pfYdMwgQt001CRhsGvX
QVKIkvipQvJZgCO8Nix7xYCukH0cI4TXD7S9BmRNMCPi74+Q1J3cDfKHCseynMNF
GzBeCDx6LW3CVfKKs0Mc2ecSl0c8gxaPDi3AfACRMefEAuVQyo82qJKjpI+O/Yik
z40C5OgK0XfetKstxcH4B0bx0o/PrUpYFM/gHHgFkbpVg5citcvFY4VcEkWryVcg
yF0qBPXP0OKBtCUU1ZGiCwRJy8iGd/dOOICcSCfMNy+jzzM3FSVzei69x7MYt3xu
IzCsmHpDvpdL7tiDDHgwajZeFFPTzf7Ic90K6TapQ3H59xPMxnL9K5o9rP1glRY0
8e4zYjYxg9A6Yl3K5zdqs+M1A3Os70HUlWZXZ4LQNcidPd1rhnPnm9eXkyV2ScXl
dE38aOA5pnrL0WZUM3/OLAToMP6h4rjw9WLqqgWlrl6yz9bhZrfRxlhZaEtNs1bi
pgrmPK/a5fK++BjMSuA94EkXTVNjKWNQBzcmrff27M1TMwN+34NWj3dk/a1gyflP
QZgK3MT+0GaMCcvy1EoZ87ffLQrWwFJOw5nT83yG7VBbuerSEk/tk30bxmYN6HzO
zvQgSjDiiH+ANXVupnzDjjBREmH6V1Hv+7Q0vrjKQHd3eYvKJpAWfFr9kO8DzKck
ZkSMj487SjlHbh33z1yupuwAtjyYQ5tN1adSlDa92t0Q08udnFDQtxXEnL6rw/Du
llEuCEVC9UYcNwwQGMsGXQBFFfj1389WHr0hkSOvyS1nPiIku5kNXDhSWq7/okTS
FwnCt+wbZa6TWbXjwKzHzu4LOarV1s8DnYHKNH6HHIqsVR2oJuIuqhyREAqjeP/T
3bQjQXV0b2dlbmVyYXRlZCBLZXkgPG1laHVsQHBoZWFhLm9yZz6JATkEEwECACMF
Al37qDICGy8HCwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRDO+i9CZ70SvqMn
CACCmdzqZW68j1E45XTHz3fvqdft6fXOyrlMuDdcH2y7Zrl5JS7PlCeHzIcsSMlH
wDYpCG8km7nwZsnWqKsOXFWq1nq/j7Kv5AzR7UmPzTw/1HFSVhIFA0ZZMHAnwp7Y
bcAT+ssvo4To9CjzRp/ZI1k26RFXPWuXETa41DBIVz13Ss4SIaf7UG9FQ55o+2BA
TP48yCQqktiWOoZ0rV1ALSFlE4Gs3UWHcYxxCABA0JB4+FuCRfB8QMreLwFb47wc
dIitbVl0mQx5IXCkqhJKqR62rRy25Put4xnPhXGtXqfoYDVYvYvlsl/FA35cX+Z1
QODnLq/jQ7ZPdaFC7cFqxztk
=RvGa
-----END PGP PRIVATE KEY BLOCK-----
Key: passphrase
Value: secret123
All I want to do is extract Key and Value pair from AWS Secrets manager and import key and later decrypt file.
As you all know JSON doesn't interpret new line characters in a multi line value so GPG import_keys fails to import private key. If I just read local file having the same private key then no problem. Please let me know if there is any workaround for this issue ?
try:
secretkey = self.get_secret(secretName)
if not secretkey:
self.logger.error("Empty secret key")
exit(0)
newdict = json.loads(secretkey)**
# newdict = ast.literal_eval(secretkey)
private_key = newdict['private_key']
# private_key = open('/home/ec2-user/GPG/test_private_key.asc').read()
passphrase = newdict['passphrase']
gpg = gnupg.GPG(gnupghome=gpgHomeDir)
import_result = gpg.import_keys(private_key)
count = import_result.count
if count == 0:
self.logger.error("Failed to import private key")
sys.exit(1)
dataPath = srcDir + "/" + self.dataSource
for root, folders, files in os.walk(dataPath):
if not files:
self.logger.info("No files found so skipping .....")
continue
for filename in folders + files:
fullpath = os.path.join(root,filename)
self.logger.info("Fullpath = {0}".format(fullpath))
out_file = "/tmp/" + filename
with open(fullpath, "rb") as f:
status = gpg.decrypt_file(f, passphrase=passphrase, output=out_file)
if status.ok:
s3Prefix = root.replace(srcDir + '/', '')
s3ObjKey = s3Prefix + "/" + filename
s3InPath = "s3://" + self.inBucketName + "/" + s3Prefix + "/" + filename
with open(out_file, "rb") as fl:
self.s3Client.upload_fileobj(fl,
self.inBucketName,
s3ObjKey
)
except Exception as e:
print(str(e))
self.logger.error(str(e))
sys.exit(1)

Answer 1

I have to use base64 format to store PGP key as follows.
import base64
import gnupg
try:
gpg = gnupg.PGP(gnupghome="/home/guest/GPG")
input_data = gpg.gen_key_input(key_type='RSA',
key_length=2048,
name_email="guest#xyz.com"
passphrase="pass123")
key = gpg.gen_key(input_data)
ascii_armored_public_key = gpg.export_keys(key.fingerprint, armor=True)
ascii_armored_private_key = gpg.export_keys(key.fingerprint, True, armor=True)
b64_encoded_private_key = base64.b64encode(ascii_armored_private_key.encode())
binaryPrivKeyFile = "/tmp/b64encoded_private_key.asc"
with open(binaryPrivKeyFile, 'wb') as bPrivFile:
bPrivFile.write(b64_encoded_private_key)
except Exception as e:
print(str(e))
sys.exit(1)
Now we have to store b64encoded_private_key.asc to AWS secrets manager as follows.
$ aws secretsmanager create-secret --name private-key --secret-binary fileb://b64encoded_private_key.asc --region us-east-1
We cannot store passphrase in the same secret so we have to create separate secret for passphrase as follows.
$ aws secretsmanager create-secret --name passwd --secret-string '{"passphrase" : "pass123"}' --region us-east-1
NOTE: The secret type for private key is binary whereas for passphrase it is plain text.
After creating secret, we can use AWS secrets manager code to get private key and passphrase. The AWS Secrets Manager code decodes private key using base64.b64decode(..) method.

Answer 2

Secrets Manager does not require you to store the data in JSON format, it can store arbitrary strings or binary data.
You could either chose to break everything up and store it in separate secrets, or use a data format that supports new lines like XML.

Answer 3

The private key which you will store, won't have special characters like '\n', '\r'.
To resolve this issue, copy the output of private_key, which will have special characters.
private_key = open('/home/ec2-user/GPG/test_private_key.asc').read()
private_key
Place this private key into your secret & get it using get_secret()
Note: you will see the additional '' in the private key which will get using load_json, to handle that you need to use private_key.replace('\n','\n')
Your code will look like below.
private_key = newdict['private_key']
private_key = private_key.replace('\n','\n')
Then you will be able to get the keys.