I am trying to figure out how to get this yubihsm2 to work with signing eth transactions. I have been using the python lib and so far i have had some basic setup. Below is an abbreviation of what i have
web3_endpoint = ''
web3 = Web3(HTTPProvider(web3_endpoint))
hsm = YubiHsm.connect("http://localhost:12345")
session = hsm.create_session_derived(1, "password")
key = session.get_object(1,OBJECT.ASYMMETRIC_KEY)
#key = AsymmetricKey.generate(session, 1, "EC Key", 1, CAPABILITY.SIGN_ECDSA, ALGORITHM.EC_K256)
pub_key = key.get_public_key()
#raw_pub = pub_key.public_bytes(
# encoding=serialization.Encoding.DER,
# format=serialization.PublicFormat.SubjectPublicKeyInfo
# )
raw_pub = pub_key.public_bytes(
encoding=serialization.Encoding.X962,
format=serialization.PublicFormat.UncompressedPoint
)
print ("Public key (Uncompressed):\n",binascii.b2a_hex(raw_pub))
unindexPub = raw_pub[1:]
public_key_hash = Web3.keccak(unindexPub)
address_bytes = public_key_hash[-20:]
address = address_bytes.hex()
print(address)
Now so far i can consistently get the same public key and it looks correct. I then get the same public key each time. When i say correct, the formatting looks correct and is the correct number of bytes.
1). should i be using the commented out public key formatting or the uncompressed X962 encoding that i have above.
From there, this is where things get a bit weird
transaction = {
'to': Web3.toChecksumAddress('0x785AB1daE1b0Ee3f2412aCF55e4153A9517b07e1'),
'gas': 21000,
'gasPrice': Web3.toWei(5, 'gwei'),
'value': 1,
'nonce': 1,
'chainId': 4,
}
serializable_transaction = serializable_unsigned_transaction_from_dict(transaction)
transaction_hash = serializable_transaction.hash()
print(transaction_hash.hex())
# sign the transaction hash and calculate v value
signature = key.sign_ecdsa(transaction_hash,hashes.SHA3_256())
r, s = ecdsa.util.sigdecode_der(signature, ecdsa.SECP256k1.generator.order())
print("r: "+str(r)+"\ns: "+str(s))
v = 28
# encode the transaction along with the full signature and send it
encoded_transaction = encode_transaction(serializable_transaction, vrs=(v, r, s))
web3.eth.sendRawTransaction(encoded_transaction)
I am settings v to 28.. i also test it with 27.. I could use the correct amount with the chainid.. but it's not necessary right from the perspective of trying to get a valid signature (recoverable to get the same public key each time). Sometimes i am getting the error "invalid sender" and other times i am getting the error "insufficient gas." If i take the signature output and use a javascript lib to try to find the public key, each time i am getting a different public key. But i keep consistently generating the same public key from the yubihsm2 in this python app.
I have also commented out in sign_ecdsa the hashing function as i am passing in the data already hashed (in order to use keccak256).
Is there something i am missing? Why are these transactions not signing correctly for eth?
i am getting some of those serialization helpers from enter link description here
helper serialization functions
Thanks
Related
The following code goes over the 10 pages of JSON returned by GET request to the URL.
and checks how many records satisfy the condition that bloodPressureDiastole is between the specified limits. It does the job, but I was wondering if there was a better or cleaner way to achieve this in python
import urllib.request
import urllib.parse
import json
baseUrl = 'https://jsonmock.hackerrank.com/api/medical_records?page='
count = 0
for i in range(1, 11):
url = baseUrl+str(i)
f = urllib.request.urlopen(url)
response = f.read().decode('utf-8')
response = json.loads(response)
lowerlimit = 110
upperlimit = 120
for elem in response['data']:
bd = elem['vitals']['bloodPressureDiastole']
if bd >= lowerlimit and bd <= upperlimit:
count = count+1
print(count)
There is no access through fields to json content because you get dict object from json.loads (see translation scheme here). It realises access via __getitem__ method (dict[key]) instead of __getattr__ (object.field) as keys may be any hashible objects not only strings. Moreover, even strings cannot serve as fields if they starts with digits or are the same with built-in dictionary methods.
Despite this, you can define your own custom class realising desired behavior with acceptable key names. json.loads has an argument object_hook wherein you may put any callable object (function or class) that take a dict as the sole argument (not only the resulted one but every one in json recursively) & return something as the result. If your jsons match some template, you can define a class with predefined fields for the json content & even with methods in order to get a robust Python-object, a part of your domain logic.
For instance, let's realise the access through fields. I get json content from response.json method from requests but it has the same arguments as in json package. Comments in code contain remarks about how to make your code more pythonic.
from collections import Counter
from requests import get
class CustomJSON(dict):
def __getattr__(self, key):
return self[key]
def __setattr__(self, key, value):
self[key] = value
LOWER_LIMIT = 110 # Constants should be in uppercase.
UPPER_LIMIT = 120
base_url = 'https://jsonmock.hackerrank.com/api/medical_records'
# It is better use special tools for handling URLs
# in order to evade possible exceptions in the future.
# By the way, your option could look clearer with f-strings
# that can put values from variables (not only) in-place:
# url = f'https://jsonmock.hackerrank.com/api/medical_records?page={i}'
counter = Counter(normal_pressure=0)
# It might be left as it was. This option is useful
# in case of additional counting any other information.
for page_number in range(1, 11):
records = get(
base_url, data={"page": page_number}
).json(object_hook=CustomJSON)
# Python has a pile of libraries for handling url requests & responses.
# urllib is a standard library rewritten from scratch for Python 3.
# However, there is a more featured (connection pooling, redirections, proxi,
# SSL verification &c.) & convenient third-party
# (this is the only disadvantage) library: urllib3.
# Based on it, requests provides an easier, more convenient & friendlier way
# to work with url-requests. So I highly recommend using it
# unless you are aiming for complex connections & url-processing.
for elem in records.data:
if LOWER_LIMIT <= elem.vitals.bloodPressureDiastole <= UPPER_LIMIT:
counter["normal_pressure"] += 1
print(counter)
I have struggle below question with days, and posted same question earlier and didn't get any positive feedback.
Im using mysql in build aes_encrypt method to encrypt new and existing data.
https://dev.mysql.com/doc/refman/8.0/en/encryption-functions.html
SET ##SESSION.block_encryption_mode = 'aes-256-ecb';
INSERT INTO test_aes_ecb ( column_one, column_two )
values ( aes_encrypt('text','key'), aes_encrypt('text', 'key'));
I used ecb ciper, so It no need to use iv for that. Issue is I can't decrypt it from node.js side.
Im using sequelize and tried to call data through model --> decrypt from node side.
I tried with below libraries,
"aes-ecb": "^1.3.15",
"aes256": "^1.1.0",
"crypto-js": "^4.1.1",
"mysql-aes": "0.0.1",
Below are code snippets from sequelize call
async function testmysqlAESModel () {
const users = await test.findAll();
console.log('users', users[0].column_one);
var decrypt = AES.decrypt( users[0].column_one, 'key' );
}
Its returning buffer data and couldn't decrypt from node side, Can someone provide proper example for that? Im struggling for days.
EDIT
Inserted record to mysql as below query.
SET ##SESSION.block_encryption_mode = 'aes-256-ecb';
INSERT INTO test_aes_ecb ( id, column_one, column_two )
VALUES (1, 2,AES_ENCRYPT('test',UNHEX('gVkYp3s6v9y$B&E)H#McQeThWmZq4t7w')));
In nodejs called like this,
testmysqlAESModel();
async function testmysqlAESModel () {
const users = await test.findAll();
console.log('users', users[0].column_one);
var decipher = crypto.createDecipheriv(algorithm, Buffer.from("gVkYp3s6v9y$B&E)H#McQeThWmZq4t7w", "hex"), "");
var encrypted = Buffer.from(users[0].column_one); // Note that this is what is stored inside your database, so that corresponds to users[0].column_one
var decrypted = decipher.update(encrypted, 'binary', 'utf8');
decrypted += decipher.final('utf8');
console.log(decrypted);
}
Im getting below error,
I used below link to create 256bit key.
https://www.allkeysgenerator.com/Random/Security-Encryption-Key-Generator.aspx
Still couldn't fix, can you provide sample project or any kind of supporting code snippet for that ?
There are multiple issues here:
Ensure that your key has the correct length. AES is specified for certain key length (i.e. 128, 196 and 256 bit). if you use any other key length, then your key will be padded (zero extended) or truncated by the crypto library. This is a non-standard process, and different implementations will do this differently. To avoid this, use a key in the correct length and store it has hex instead of ascii (to avoid charset issues)
Potential issues regarding password to key inference. Some AES implementations use methods to infer keys from passwords/passphrases. Since you are using raw keys in MySQL, you do not want to infer anything but want to use raw keys in NodeJS as well. This means that if you are using the native crypto module, that you want to use createDecipheriv instead of createDecipher.
Caution: The AES mode you are using (ECB) is inherently insecure, because equal input leads to equal output. There are ways around that using other AES modes, such as CBC or GCM. You have been warned.
Example:
MySQL SELECT AES_ENCRYPT('text',UNHEX('F3229A0B371ED2D9441B830D21A390C3')) as test; returns the buffer [145,108,16,83,247,49,165,147,71,115,72,63,152,29,218,246];
Decoding this in Node could look like this:
var crypto = require('crypto');
var algorithm = 'aes-128-ecb';
var decipher = crypto.createDecipheriv(algorithm, Buffer.from("F3229A0B371ED2D9441B830D21A390C3", "hex"), "");
var encrypted = Buffer.from([145,108,16,83,247,49,165,147,71,115,72,63,152,29,218,246]); // Note that this is what is stored inside your database, so that corresponds to users[0].column_one
var decrypted = decipher.update(encrypted, 'binary', 'utf8');
decrypted += decipher.final('utf8');
console.log(decrypted);
This prints text again.
Note that F3229A0B371ED2D9441B830D21A390C3 is the key in this example, you would obviously have to create your own. Just ensure that your key has the same length as the example, and is a valid hex string.
I have three float arrays of data (A,B,C) that I want to send it to event hub.
The issue here is that the classes are nested and I can't figure out how to shape the data to be able to send successfully.
Here is the code that I am trying
logger = logging.getLogger("azure")
ADDRESS = ""
USER = ""
KEY = ""
try:
if not ADDRESS:
raise ValueError("No EventHubs URL supplied.")
# Create Event Hubs client
client = EventHubClient(ADDRESS, debug=False, username=USER, password=KEY)
sender = client.add_sender(partition="0")
client.run()
x_value = np.arange(100)
try:
start_time = time.time()
for i in range(100000):
A = np.asarray([1,2,3,4])
B = np.asarray([2,3,4,5])
C = np.asarray([3,4,5,6])
message = [A, B, C]
sender.send(EventData(body = message))
time.sleep(1)
except:
raise
finally:
end_time = time.time()
client.stop()
run_time = end_time - start_time
logger.info("Runtime: {} seconds".format(run_time))
except KeyboardInterrupt:
pass
In this way, I am seeing the error 'ValueBody' object has no attribute 'append'
I think by types, the class encodes the message differently.
Instead of sending a single message with a serial string, I want to send the message in a parallel way, and receive them something like this:
Receive one eventdata with the form of the list [A,B,C]
Receive three eventdata A, B, C separately, like calling three different objects basis.
For the second way, I am not sure if it would work since I am using only one partition and that may have the three eventdata A, B, C mixed when I receive them and I do not want that.
I have confirmed with ms support, the feedback is that: it only works for a list of string or bytes, but does not work for a list of int.
And also you should note that, even it sends a list of string, when receive the list, it will concatenate the all elements of the list to compose a single message. For example, if there is a list ["a","b","c","d"] for sending, when receive it, it will show this "abcd" as a whole string.
I have tested it by myself, the above information is correct.
So for your case, you should consider change the list to a string or json string. when receiving it, you can format the data to what you need.
Hope it helps.
In Elm, you can use the Json.decode and Http package to request json data. My attempt was to work out a periodic lookup for emails ( from this url ). The timer operation does work (i tried it with with a simple counter).
I have used this example and this SO question as reference.
Now the types:
type alias Email = { title: String, ... }
type Action =
NoAction
| TickCounter -- TODO rem
| AddEmails (List Email)
Then the main + state + actions ...
main: Signal Html
main = Signal.map (view actions.address) state
state: Signal Model
state = Signal.foldp update makeEmptyModel input
-- handle inputs (merging signals)
input : Signal Action
input =
Signal.mergeMany
[ actions.signal
-- , other actions
, Signal.map checkForNewMails (Time.every (Time.minute / 6.0) ) -- TODO precise timer (quick test)
]
actions: Signal.Mailbox Action
actions = Signal.mailbox NoAction
update: Action -> Model -> Model
update action model =
case action of
NoAction -> model
TickCounter -> { model | count = model.count + 1 }
AddEmails newMails -> { model | emails = newMails }
checkForNewMails: Time -> Action
checkForNewMails t =
let mails = startGettingEmailData
in TickCounter -- TODO replace with AddEmails using mails
The TickCounter is an Action, which i have used to test my timing operation. But the problem is startGettingEmailData. It uses the next snippet, but it doesn't fire any JSON request (i have checked it through the console). Once that has been resolved, i can convert mails to an action so that i can add the emails in the model.
-- url to json
jsonUrl = "https://dl.dropboxusercontent.com/u/14070433/temp.json"
-- get emails
startGettingEmailData: Task Http.Error (List Email)
startGettingEmailData = Http.get emailJsonDecoder jsonUrl
emailJsonDecoder: Json.Decoder (List Email)
emailJsonDecoder =
let makeEmail = Json.object4
(\ti fr da bo -> makeNewEmail -1 ti fr da bo )
("title" := Json.string)
("from" := Json.string)
("date" := Json.string)
("body" := Json.string)
in
"emails" := Json.list makeEmail
Is there a problem with my code ? If not, then is there a way to check the Http.Error content ? (Maybe the problem lies not in my code, but in the network, but i can access the dropbox file by browser...)
Your code for checkForNewMails doesn't actually do anything with mails, so it never gets invoked. The let statement doesn't make any calls, it only lets you define one-off functions within the body of a larger function. Since the in portion merely returns TickCounter, then it means this function only ever returns TickCounter and does nothing else.
Furthermore, startGettingEmailData is returning a Task, which means it only gets invoked when in a port. You can't use it in a function that only returns an Action, because it would never get run.
You'll instead want to write a port that triggers on a timer, then creates a Task which polls your url, then maps the response of that GET request to an Action, calling the actions mailbox. You can use Task.onError to write a simple error handler, which could forward an error message to your view by creating an Error String constructor on your Action type.
Here's an example:
getEmailData _ =
let
request =
Http.get emailJsonDecoder jsonUrl
|> Task.map AddEmails
in
request
`Task.onError` (\err -> Task.succeed (Error (toString err)))
`Task.andThen` (\action -> Signal.send actions.address action)
port getEmails : Signal (Task a ())
port getEmails =
Signal.map getEmailData (Time.every (Time.minute / 6.0) )
The above code will cause the URL to be retrieved, parsed, then, on success, it will trigger you actions mailbox and cause an update of the view. If there is an error, it will send the new action of Error with a message, which you should handle in the view. (You'll have to add Error String to the Action union type and handle it in the model, update, and view functions).
This has become more of an exercise in what am I doing wrong than mission critical, but I'd still like to see what (simple probably) mistake I'm making.
I'm using mysql (5.1.x) AES_ENCRYPT to encrypt a string. I'm using CF's generateSecretKey('AES') to make a key (I've tried it at defaul and 128 and 256 bit lengths).
So let's say my code looks like this:
<cfset key = 'qLHVTZL9zF81kiTnNnK0Vg=='/>
<cfset strToEncrypt = '4111111111111111'/>
<cfquery name="i" datasource="#dsn#">
INSERT INTO table(str)
VALUES AES_ENCRYPT(strToEncrypt,'#key#');
</cfquery>
That works fine as expected and I can select it using SELECT AES_DECRYPT(str,'#key#') AS... with no problems at all.
What I can't seem to do though is get CF to decrypt it using something like:
<cfquery name="s" datasource="#dsn#">
SELECT str
FROM table
</cfquery>
<cfoutput>#Decrypt(s.str,key,'AES')#</cfoutput>
or
<cfoutput>#Decrypt(toString(s.str),key,'AES')#</cfoutput>
I keep getting "The input and output encodings are not same" (including the toString() - without that I get a binary data error). The field type for the encrypted string in the db is blob.
This entry explains that mySQL handles AES-128 keys a bit differently than you might expect:
.. the MySQL algorithm just or’s the bytes of a given passphrase
against the previous bytes if the password is longer than 16 chars and
just leaves them 0 when the password is shorter than 16 chars.
Not highly tested, but this seems to yield the same results (in hex).
<cfscript>
function getMySQLAES128Key( key ) {
var keyBytes = charsetDecode( arguments.key, "utf-8" );
var finalBytes = listToArray( repeatString("0,", 16) );
for (var i = 1; i <= arrayLen(keyBytes); i++) {
// adjust for base 0 vs 1 index
var pos = ((i-1) % 16) + 1;
finalBytes[ pos ] = bitXOR(finalBytes[ pos ], keyBytes[ i ]);
}
return binaryEncode( javacast("byte[]", finalBytes ), "base64" );
}
key = "qLHVTZL9zF81kiTnNnK0Vg==";
input = "4111111111111111";
encrypted = encrypt(input, getMySQLAES128Key(key), "AES", "hex");
WriteDump("encrypted="& encrypted);
// note: assumes input is in "hex". either convert the bytes
// to hex in mySQL first or use binaryEncode
decrypted = decrypt(encrypted, getMySQLAES128Key(key), "AES", "hex");
WriteDump("decrypted="& decrypted);
</cfscript>
Note: If you are using mySQL for encryption be sure to see its documentation which mentions the plain text may end up in various logs (replication, history, etectera) and "may be read by anyone having read access to that information".
Update: Things may have changed, but according to this 2004 bug report the .mysql_history file is only on Unix. (Keep in mind there may be other log files) Detailed instructions for clearing .mysql_history can be found in the manual, but in summary:
Set the MYSQL_HISTFILE variable to /dev/null (on each log in)
Create .mysql_history as a symbolic link to /dev/null (only once)