Couchbase Document ID Requirements and limitations - couchbase

What are the restrictions for a Couchabse Document's ID string?
Length?
Are special characters allowed?
What does the string have to start and end with?
Couchbase Guide Sample Code:
var properties = new Dictionary<string, object>
{
{"title", "Little, Big"},
{"author", "John Crowley"},
{"published", 1982}
};
var document = database.GetDocument("978-0061120053");
Debug.Assert(document != null);
var rev = document.PutProperties(properties);
On var document = database.GetDocument("978-0061120053"); what can be used in place of "978-0061120053"?


Quoting from the Couchbase Developer guide these are the only limits on keys:
Keys are strings, typically enclosed by quotes for any given SDK.
No spaces are allowed in a key.
Separators and identifiers are allowed, such as underscore: ‘person_93847’.
A key must be unique within a bucket; if you attempt to store the same key in a bucket, it will either overwrite the value or return an
error in the case of add().
Maximum key size is 250 bytes. Couchbase Server stores all keys in RAM and does not remove these keys to free up space in RAM. Take this
into consideration when you select keys and key length for your
application.

Related

convert FiX message to JSON using JAVA

i have extracted the fix message as below from Unix server and now need to convert this message into JSON. how can we do this?
8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111

Note: I tried to make this a comment on the answer provided by #selbie, but the text was too long for a comment, so I am making it an answer.
#selbie's answer will work most of the time, but there are two edge cases in which it could fail.
First, in a tag=value field where the value is of type STRING, it is legal for value to contain the = character. To correctly cope with this possibility, the Java statement:
pair = item.split("=");
should be changed to:
pair = item.split("=", 2);
The second edge case is when there are a pair of fields, the first of which is of type LENGTH and the second is of type DATA. In this case, the value of the LENGTH fields specifies the length of the DATA field (without the delimiter), and it is legal for the value of the DATA field to contain the delimiter character (ASCII character 1, but denoted as | in both the question and Selbie's answer). Selbie's code cannot be modified in a trivial manner to deal with this edge case. Instead, you will need a more complex algorithm that consults a FIX data dictionary to determine the type of each field.

Since you didn't tag your question for any particular programming language, I'll give you a few sample solutions:
In javascript:
let s = "8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111"
let obj = {};
items = s.split("|")
items.forEach(item=>{
let pair = item.split("=");
obj[pair[0]] = pair[1];
});
let jsonString = JSON.stringify(obj);
Python:
import json
s = "8=FIXT.1.1|9=449|11=ABCD1|35=AE|34=1734|49=REPOFIXUAT|52=20140402-11:38:34|56=TR_UAT_VENDOR|1128=8|15=GBP|31=1.7666|32=50000000.00|55=GBP/USD|60=20140402-11:07:33|63=B|64=20140415|65=OR|75=20140402|150=F|167=FOR|194=1.7654|195=0.0012|460=4|571=7852455|1003=2 USD|1056=88330000.00|1057=N|552=1|54=2|37=20140402-12:36:48|11=NOREF|453=4|448=ZERO|447=D|452=3|448=MBY2|447=D|452=1|448=LMEB|447=D|452=16|448=DOR|447=D|452=11|826=0|78=1|79=default|80=50000000.00|5967=88330000.00|10=111"
obj = {}
for item in s.split("|"):
pair = item.split("=")
obj[pair[0]] = pair[1]
jsonString = json.dumps(obj)
Porting the above solutions to other languages is an exercise for yourself. There's comments below about semantic ordering and handling cases where the the = or | chars are part of the content. That's on you to explore if you need to support those scenarios.

AES encryption on mysql and node.js

I have struggle below question with days, and posted same question earlier and didn't get any positive feedback.
Im using mysql in build aes_encrypt method to encrypt new and existing data.
https://dev.mysql.com/doc/refman/8.0/en/encryption-functions.html
SET ##SESSION.block_encryption_mode = 'aes-256-ecb';
INSERT INTO test_aes_ecb ( column_one, column_two )
values ( aes_encrypt('text','key'), aes_encrypt('text', 'key'));
I used ecb ciper, so It no need to use iv for that. Issue is I can't decrypt it from node.js side.
Im using sequelize and tried to call data through model --> decrypt from node side.
I tried with below libraries,
"aes-ecb": "^1.3.15",
"aes256": "^1.1.0",
"crypto-js": "^4.1.1",
"mysql-aes": "0.0.1",
Below are code snippets from sequelize call
async function testmysqlAESModel () {
const users = await test.findAll();
console.log('users', users[0].column_one);
var decrypt = AES.decrypt( users[0].column_one, 'key' );
}
Its returning buffer data and couldn't decrypt from node side, Can someone provide proper example for that? Im struggling for days.
EDIT
Inserted record to mysql as below query.
SET ##SESSION.block_encryption_mode = 'aes-256-ecb';
INSERT INTO test_aes_ecb ( id, column_one, column_two )
VALUES (1, 2,AES_ENCRYPT('test',UNHEX('gVkYp3s6v9y$B&E)H#McQeThWmZq4t7w')));
In nodejs called like this,
testmysqlAESModel();
async function testmysqlAESModel () {
const users = await test.findAll();
console.log('users', users[0].column_one);
var decipher = crypto.createDecipheriv(algorithm, Buffer.from("gVkYp3s6v9y$B&E)H#McQeThWmZq4t7w", "hex"), "");
var encrypted = Buffer.from(users[0].column_one); // Note that this is what is stored inside your database, so that corresponds to users[0].column_one
var decrypted = decipher.update(encrypted, 'binary', 'utf8');
decrypted += decipher.final('utf8');
console.log(decrypted);
}
Im getting below error,
I used below link to create 256bit key.
https://www.allkeysgenerator.com/Random/Security-Encryption-Key-Generator.aspx
Still couldn't fix, can you provide sample project or any kind of supporting code snippet for that ?

There are multiple issues here:
Ensure that your key has the correct length. AES is specified for certain key length (i.e. 128, 196 and 256 bit). if you use any other key length, then your key will be padded (zero extended) or truncated by the crypto library. This is a non-standard process, and different implementations will do this differently. To avoid this, use a key in the correct length and store it has hex instead of ascii (to avoid charset issues)
Potential issues regarding password to key inference. Some AES implementations use methods to infer keys from passwords/passphrases. Since you are using raw keys in MySQL, you do not want to infer anything but want to use raw keys in NodeJS as well. This means that if you are using the native crypto module, that you want to use createDecipheriv instead of createDecipher.
Caution: The AES mode you are using (ECB) is inherently insecure, because equal input leads to equal output. There are ways around that using other AES modes, such as CBC or GCM. You have been warned.
Example:
MySQL SELECT AES_ENCRYPT('text',UNHEX('F3229A0B371ED2D9441B830D21A390C3')) as test; returns the buffer [145,108,16,83,247,49,165,147,71,115,72,63,152,29,218,246];
Decoding this in Node could look like this:
var crypto = require('crypto');
var algorithm = 'aes-128-ecb';
var decipher = crypto.createDecipheriv(algorithm, Buffer.from("F3229A0B371ED2D9441B830D21A390C3", "hex"), "");
var encrypted = Buffer.from([145,108,16,83,247,49,165,147,71,115,72,63,152,29,218,246]); // Note that this is what is stored inside your database, so that corresponds to users[0].column_one
var decrypted = decipher.update(encrypted, 'binary', 'utf8');
decrypted += decipher.final('utf8');
console.log(decrypted);
This prints text again.
Note that F3229A0B371ED2D9441B830D21A390C3 is the key in this example, you would obviously have to create your own. Just ensure that your key has the same length as the example, and is a valid hex string.

iterate through Poco::JSON::Object in insertion order

It is possible to preserve insertion order when parsing a JSON struct with a
Poco::JSON::Parser( new Poco::JSON::ParseHandler( true ) ): the non-default ParseHandler parameter preserveObjectOrder = true is handed over to the Poco::JSON::Objects so that they keep an private list of keys sorted in insertion order.
An object can then be serialized via Object::stringify() to look just like the source JSON string. Fine.
What, however, is the official way to step through a Poco::JSON::Object and access its internals in insertion order? Object::getNames() and begin()/end() use the alphabetical order of keys, not insertion order -- is there another way to access the values, or do I have to patch Poco?

As you already said:
Poco::JSON::ParseHandler goes into the Poco::JSON::Parser-constructor.
Poco::JSON::Parser::parse() creates a Poco::Dynamic::Var.
From that you'll extract a Poco::JSON::Object::Ptr.
The Poco::JSON:Object has the method "getNames". Beginning with this commit it seems to preserve the order, if it was requested via the ParseHandler. (Poco::JSON:Object::getNames 1.8.1, Poco::JSON:Object::getNames 1.9.0)
So now it should work as expected to use:
for(auto const & name : object->getNames()){
auto const & value = object->get(name); // or one of the other get-methods
// ... do things ...
}

Setting lua table in redis

I have a lua script, which simplified is like this:
local item = {};
local id = redis.call("INCR", "counter");
item["id"] = id;
item["data"] = KEYS[1]
redis.call("SET", "item:" .. id, cjson.encode(item));
return cjson.encode(item);
KEYS[1] is a stringified json object:
JSON.stringify({name : 'some name'});
What happens is that because I'm using cjson.encode to add the item to the set, it seems to be getting stringified twice, so the result is:
{"id":20,"data":"{\"name\":\"some name\"}"}
Is there a better way to be handling this?

First, regardless your question, you're using KEYS the wrong way and your script isn't written according to the guidelines. You should not generate key names in your script (i.e. call SET with "item:" .. id as a keyname) but rather use the KEYS input array to declare any keys involved a priori.
Secondly, instead of passing the stringified string with KEYS, use the ARGV input array.
Thirdly, you can do item["data"] = json.decode(ARGV[1]) to avoid the double encoding.
Lastly, perhaps you should learn about Redis' Hash data type - it may be more suitable to your needs.

Storing items by an ID in an array

I have numerous items that need to be stored and retrieved by their ID. Their IDs, however, do not always start at zero. In fact, they may be much higher, such as 500 or more.
If I store these in an array, so array[0] -> array[499] are null and then array[500] -> array[500+n] contain the objects, is this going to affect performance? Alternatively, would it be better storing them in array[0] -> array[n] and iterating though the list until I find the item with the corresponding ID?
Thanks,
Will

Without knowing how you plan on using your array, from the brief description you've given, I would suggest using a Dictionary instead.
var dict:Dictionary = new Dictionary();
var someObject:Object {id: 500};
dict[someObject.id] = someObject; // store someObject in key 500
var retrievedObject:Object = dict[500]; // retrieve object from key 500