Store Series of 0s and 1s in MySQL database - mysql

I'm using MySql and Nodejs. I need to store a huge string made of only 0s and 1s. (This is a map grid. 0 = Can Move, 1 = Can't move.)
Ex: 00000000000000000000000000000000000010000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000010000000000
...
Should I take that string, encode it into a base64 string and store it that way? Then when I need it, I decode it? If so, how would I do it, using Nodejs/Javascript?
Thanks

The data you're storing appears to compress easily, why don't you use the built-in MySql COMPRESS function
Raw:
mysql> select length('00000000000000000000000000000000000010000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000010000000000');
332
With COMPRESS
mysql> select length(compress('00000000000000000000000000000000000010000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000010000000000'));
23

Don't store a string, store binary.
Here's an example of a way you could convert that data to binary. Hopefully you can reengineer things so that your map data is never represented as a string in the first place.
static void Main(string[] args)
{
const string data =
"00000000000000000000000000000000000010000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000100000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000010000000000";
var arrayLength = data.Length/8;
if (data.Length%8 != 0)
arrayLength++;
var binaryData = new byte[arrayLength];
byte nextByte=0;
var k = -1;
for (var i = 0; i < data.Length; i++)
{
var j = i%8;
if (j == 0)
{
binaryData[++k] = nextByte;
nextByte = 0;
}
var bit = int.Parse(data[i].ToString());
if(bit==1)
nextByte |= (byte)(bit << j);
}
binaryData[k] = nextByte;
}

Related

Autokey Encryption

I am working on a project to write to and read from a TP Link / Kaza power strip or smart plug.
The data that is sent is encrypted json that has been "autokey encrypted".
So far I have been able to convert a typescript encrypt function and it works well. I get the expected result. However, I need to add a "header" to my encrypted data. That data is 3 null bytes followed by a byte that is a measure of the length of the encrypted bytes.
The typescript example has this bit of code to "encrypt with headers", however, I've hit a bit of a wall trying to convert it to something usable. Can someone nudge me along the path ?
First are the two typescript functions: (borrowed from https://github.com/plasticrake/tplink-smarthome-crypto/blob/master/src/index.ts)
/**
* Encrypts input where each byte is XOR'd with the previous encrypted byte.
*
* #param input - Data to encrypt
* #param firstKey - Value to XOR first byte of input
* #returns encrypted buffer
*/
export function encrypt(input: Buffer | string, firstKey = 0xab): Buffer {
const buf = Buffer.from(input);
let key = firstKey;
for (let i = 0; i < buf.length; i += 1) {
// eslint-disable-next-line no-bitwise
buf[i] ^= key;
key = buf[i];
}
return buf;
}
/**
* Encrypts input that has a 4 byte big-endian length header;
* each byte is XOR'd with the previous encrypted byte.
*
* #param input - Data to encrypt
* #param firstKey - Value to XOR first byte of input
* #returns encrypted buffer with header
*/
export function encryptWithHeader(
input: Buffer | string,
firstKey = 0xab
): Buffer {
const msgBuf = encrypt(input, firstKey);
const outBuf = Buffer.alloc(msgBuf.length + 4);
outBuf.writeUInt32BE(msgBuf.length, 0);
msgBuf.copy(outBuf, 4);
return outBuf;
}
Second is what I have so far.
// This part works well and produces the expected results
String encrypt(String input)
{
int16_t firstKey = 0xab;
String buf;
int key;
int i;
buf = input;
key = firstKey;
i = 0;
for (;i < buf.length();(i = i + 1))
{
buf[i] ^= key;
key = buf[i];
}
return buf;
}
// This does not function yet, as I'm pretty lost..
// This was orginally converted from typescript with https://andrei-markeev.github.io/ts2c/
// I started work on converting this, but ran into errors I don't know how to solve.
String encryptWithHeader(String input){
String msgBuf;
String outBuf;
int16_t firstKey = 0xab;
char * null = NULL;
msgBuf = encrypt(input);
outBuf = msgBuf.length() +1;
//this is where I got lost...
assert(null != NULL);
null[0] = '\0';
strncat(null, outBuf, msgBuf.length());
str_int16_t_cat(null, 4);
outBuf = msgBuf + 4
return outBuf;
}
Finally, the data:
//this is the unencrypted json
String offMsg = "{\"system\":{\"set_relay_state\":{\"state\":0}}}";
//current encrypt function produces:
d0f281f88bff9af7d5ef94b6c5a0d48bf99cf091e8b7c4b0d1a5c0e2d8a381f286e793f6d4eedea3dea3
//the working "withheaders" should produce:
00002ad0f281f88bff9af7d5ef94b6c5a0d48bf99cf091e8b7c4b0d1a5c0e2d8a381f286e793f6d4eedea3dea3
Admittedly my C/C++ ability is very limited and I can spell typescript, that's about all. I have a very extensive history with PHP. As useful as that is. So, I understand the basics of data structures and whatnot, but I'm venturing off into areas I've never been in. Any help would be greatly appreciated.
It looks like the encryption is fairly simple: write the current character XORed with the key to the buffer and make that newly written character the new key. It also looks like the "withHeaders" version adds the length of the encrypted string as a 4 byte integer to the start of the buffer. I think it might be easier to allocate a character array and pass that array to a function that writes the result to that buffer. For example:
void encryptWithHeader(byte buffer[], int bufferLength, byte key, String message) {
int i;
uint32_t messageLength = message.length();
Serial.println(message);
Serial.println(message.length());
// check that we won't overrun the buffer
if ( messageLength + 5 < bufferLength) {
buffer[0] = messageLength >> 24 & 0xFF;
buffer[1] = messageLength >> 16 & 0xFF;
buffer[2] = messageLength >> 8 & 0xFF;
buffer[3] = messageLength & 0xFF;
for (i = 0; i < messageLength; i++) {
buffer[i + 4] = message[i] ^ key;
key = buffer[i + 4];
}
}
else { // we would have overrun the buffer
Serial.println("not enough room in buffer for message");
}
}
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
}
void loop() {
byte theBuffer[64];
int i;
String offMsg = "{\"system\":{\"set_relay_state\":{\"state\":0}}}";
encryptWithHeader(theBuffer, 64, 0xab, offMsg);
// now print it out to check
for (i = 0; i < offMsg.length() + 4; i++) {
if (theBuffer[i] < 0x10) // adds an extra zero if a byte prints as on1y 1 char
Serial.print("0");
Serial.print(theBuffer[i], HEX);
}
while (true)
;
}
If you want to send the character buffer to a remote device you can send it out one byte at a time:
for (i = 0; i < offMsg.length() + 4; i++)
Serial.write(theBuffer[i]);

How to write a sequence of bytes to binary file using JScript?

I'm trying to write Array(1, 2) to binary file as bytes.
So output file should contain 00000001 00000010.
I understood that I have to use ADODB.Stream, but I haven't found any solution for such simple task.
var data = new Array(1, 2)
out = WScript.CreateObject("ADODB.Stream")
out.Type = 1
out.Open()
out.Write(data)
out.SaveToFile("output.bin", 2)
out.Close()
Code above gives error:
Arguments are of the wrong type, are out of acceptable range, or are in conflict with one another.
Any ideas how to convert data array to acceptable type?
You can use the WriteAll function described here: Reading and Writing Binary Files Using JScript. Copyright: Dr Alexander J Turner.
Please note, the Stream shall be of type text, and then you will need to do the text<->binary conversion.
I just tested all the code and it worked fine for me.
Here is a complete example:
var bf1 = new BinaryFile("C:/Temp/test.bin");
var outBuf = '';
for(var i=0, l=data.length; i<l; i++) {
outBuf += String.fromCharCode(data[i]);
}
bf1.WriteAll(outBuf);
The result looks like this:
EDIT:
I just did a more compact adaptation of the code, to avoid the hex conversion and the double loop:
// Codepage conversion table
var _c=[199,252,233,226,228,224,229,231,234,235,232,239,238,236,196,197,
201,230,198,244,246,242,251,249,255,214,220,162,163,165,8359,402,
225,237,243,250,241,209,170,186,191,8976,172,189,188,161,171,187,
9617,9618,9619,9474,9508,9569,9570,9558,9557,9571,9553,9559,9565,9564,9563,9488,
9492,9524,9516,9500,9472,9532,9566,9567,9562,9556,9577,9574,9568,9552,9580,9575,
9576,9572,9573,9561,9560,9554,9555,9579,9578,9496,9484,9608,9604,9612,9616,9600,
945,223,915,960,931,963,181,964,934,920,937,948,8734,966,949,8745,
8801,177,8805,8804,8992,8993,247,8776,176,8729,183,8730,8319,178,9632,160],
_w=[],
_r=[];
// Create forward lookup to write & reverse lookup to read
for(var i=0, l=256; i<l; i++) {
var c = (i<128) ? i : _c[i-128];
_w[i] = c;
_r[c] = i;
}
// Read binary data from disk
function binFileToArray(fileName, binArray){
var inStream = new ActiveXObject("ADODB.Stream");
inStream.Type = 2;
inStream.CharSet = '437';
inStream.Open();
inStream.LoadFromFile(fileName);
var inString = inStream.ReadText();
inStream.Close();
for(var i=0, l=inString.length; i<l; i++) {
binArray.push(_r[inString.charCodeAt(i)]);
}
}
// Write binary data to disk
function binArrayToFile(binArray, fileName){
var outStream = new ActiveXObject('ADODB.Stream');
var outString = '';
for(var i=0, l=binArray.length; i<l; i++) {
outString += String.fromCharCode(_w[binArray[i]]);
}
outStream.Type = 2;
outStream.CharSet = '437';
outStream.Open();
outStream.WriteText(outString);
outStream.SaveToFile(fileName, 2);
outStream.Close();
}
// TEST: read binary file from disk & write array back to disk
var testArray = [];
binFileToArray('c:/temp/test.bin', testArray);
binArrayToFile(testArray, 'c:/temp/test2.bin');
// both files equals

Flash Builder 4.6 | mx:DataGrid | Decimal to Fraction...or Fraction to Decimal Formatting

My datagrid has a depth-dimensions column that shows fractions (one dimension used for example is 24 3/8). I have the ability to choose the text (fraction or decimal), but essentially I would need to be able to convert back and forth from 24 3/8 to 24.375.
Why the Decimal format is needed:
I have checkboxes to filter the depth-dimensions column, so I'll need decimal form for that logic (e.g. checkbox to see filter depth-dimensions between 20 and 26).
Why the fraction format is needed: I'll need the fraction format because that depth-dimension data will be referenced as a string in another part of the application. The filter doesn't work when in this format is used in the dataGrid, because it won't recognize 24 3/8 as a number/int.
So basically I'm looking for a way to convert between the two formats, 24 3/8 to 24.375 and 24.375 to 24 3/8.
Again, my apologies for the confusion - I'm able to re-edit and/or add more details if needed.
Thanks in advance!
--moe
Why the fraction format is needed: I'll need the fraction format
because that depth-dimension data will be referenced as a string in
another part of the application.
Your reason for needing the fraction format seems odd. Do you understand that you can use a Number data type in a String by casting it?
var decimalNum:Number = 3.14;
//concatenating a Number with a String automatically casts it
var autoCastString:String = "I want to eat some " + decimalNum;
trace(autoCastString);
// cast as String type
var decimalString:String = String(decimalNum);
trace("Mmmm! I like", decimalString);
Output:
I want to eat some 3.14
Mmmm! I like 3.14
But perhaps you have other reasons. The code below is from this link: Decimal to Fraction. I haven't tested it.
package com.lookmum.util
{
public class Fraction
{
private static var it :Number = 0;
public static var iterationLimit:Number = 10000;
public static var accuracy :Number = 0.00001;
public function Fraction()
{
}
private static function resetIt():void
{
it = 0;
}
private static function addIt():Boolean
{
it++;
if (it == iterationLimit)
{
trace('error : too many iterations');
return true;
}
else
{
return false;
}
}
public function getFractionString(num:Number):String
{
var fracString:String;
var fracArray:Array = getFraction(num);
switch (fracArray.length)
{
case 1 :
fracString = num.toString();
break;
case 2 :
fracString = fracArray[0].toString() + '/' + fracArray[1].toString();
break;
case 3 :
fracString = fracArray[0].toString() + ' ' + fracArray[1].toString() + '/' + fracArray[2].toString();
break;
}
return fracString;
}
public function getFraction(num:Number):Array
{
var fracArray:Array = new Array();
var hasWhole:Boolean = false;
if (num >= 1)
{
hasWhole = true;
fracArray.push(Math.floor(num));
}
if (num - Math.floor(num) == 0)
{
return fracArray;
}
if (hasWhole)
{
num = num - Math.floor(num);
}
var a:Number = num - int(num);
var p:Number = 0;
var q:Number = a;
resetIt();
while (Math.abs(q - Math.round(q)) > accuracy)
{
addIt();
p++;
q = p / a;
}
fracArray.push(Math.round(q * num));
fracArray.push(Math.round(q));
return fracArray;
}
}
}

Improve function speed that uses remainder:

Please have a look at this piece of code:
public static function getCharLocationById(id:int):Point {
var lx:int = id % 16;
var ly:int = id / 16;
return new Point(lx, ly);
}
It works perfectly but is very slow. Does anyone know of a way to make it much faster?
Thanks in advance!
If you create the objects beforehand for all possibilities, all you have to do is look them up in an array (with the id as index).
private static const _locationLookUpTable:Array = []; //or Vector, if you like
// fill the array somewhere, maybe like this
for (var i:uint = 0; i <= maximumId; ++i) _locationLookUpTable.push(i % 16, i / 16);
public static function getCharLocationById(id:int):Point {
return _locationLookUpTable[id];
}
If the number of ids is not limited or very large you can employ an object pool.
This requires a little more code as you should return the objects to the pool if they are not used any more.
Ignore the variable creations, only takes time to create, assign and then again read them to submit them to the Point constructor.
public static function getCharLocationById(id:int):Point
{
return new Point(id % 16, id / 16);
}
Also, considering that your input is an integer, you can use bitshifts for the division by 16 like this:
id = id >> 1; // div. by 2 = id/2
id = id >> 1; // div. by 2 = id/2/2 = id/4
id = id >> 1; // div. by 2 = id/2/2/2 = id/8
id = id >> 1; // div. by 2 = id/2/2/2/2 = id/16
Shortening that we get
id = id >> 4; // (1+1+1+1 = 4)
Keep in mind that the result will also be an integer, so 11 >> 1 will return 5 and not 5.5.

Converting 2x32 bit uint <-> Number

How would one convert from 2x32bit uints to a Number and back (assume max value of 2^52)?
I believe the following would theoretically work (passing around as ByteArray for clarity, but an Array could work as storage as well), But it doesn't because bitwise operators evidently force Number into 32 bits :\
(see: Binary math on Number objects limited to 32 bits?):
public static function read64BitNumberFromBuffer(buffer:ByteArray):Number {
var ch1:uint = buffer.readUnsignedInt();
var ch2:uint = buffer.readUnsignedInt();
var num:Number = ((ch1 << 32) | ch2);
return(num);
}
public static function write64BitNumberToBuffer(num:Number):ByteArray {
var ch1:uint = uint((num & 0xFFFFFFFF00000000) >> 32);
var ch2:uint = uint(num & 0xFFFFFFFF);
var buffer:ByteArray = new ByteArray();
buffer.writeUnsignedInt(ch1);
buffer.writeUnsignedInt(ch2);
return(buffer);
}
One could use a library like as3crypto's BigInteger to handle this, but that seems like an awful lot of bloat for such a discrete need. Is there a robust bit of code that could be injected into the above functions to make them return the correct values?
Although I'd prefer a pure Actionscript solution, as a point of interest- are bitwise operators in Crossbridge also limited to 32 bits? (btw- I need 1500 reputation to create a tag "crossbridge", can someone do it on my behalf?)
EDIT: Tried readDouble()/writeDouble() as well but it seemed to want to switch to reverse the bytes for some reason under a more thorough test (tried playing with endian setting, to no avail other than it did affect output in the wrong way)
OK- this seems to work perfectly:
package
{
import flash.display.Sprite;
import flash.utils.ByteArray;
public class TEMP extends Sprite
{
public function TEMP()
{
var targetNumber:Number = 6697992365;
var buffer:ByteArray = new ByteArray();
var testNumber:Number;
write64BitNumberToBuffer(buffer, targetNumber);
buffer.position = 0;
testNumber = read64BitNumberFromBuffer(buffer);
if(targetNumber == testNumber) {
trace("Passed! Both numbers are", targetNumber);
} else {
trace("Failed! Test number is", testNumber, "When it should be", targetNumber);
}
}
public static function read64BitNumberFromBuffer(buffer:ByteArray):Number {
var finalNumber:Number;
var str:String = '';
var byte:uint;
var chr:String;
while(str.length < 16) {
byte = buffer.readUnsignedByte();
chr = byte.toString(16);
if(chr.length == 1) {
chr = '0' + chr;
}
str += chr;
}
finalNumber = Number('0x' + str);
return(finalNumber);
}
public static function write64BitNumberToBuffer(buffer:ByteArray, num:Number) {
var hexString:String = num.toString(16);
var idx:uint = 16 - hexString.length;
var byte:uint;
while(idx--) {
hexString = '0' + hexString;
}
for(idx = 0; idx < hexString.length; idx += 2) {
byte = uint('0x' + hexString.substr(idx, 2));
buffer.writeByte(byte);
}
}
}
}
Output: Passed! Both numbers are 6697992365