I'm writing an extension that makes use of the chrome.storage API. I want to truncate each item to make sure it is below the maximum bytes threshold of the storage (local and sync).
The documentation states that the byte size of each individual item is
measured by the JSON stringification of its value plus its key length.
I use the following code to calculate the expected byte size:
new TextEncoder().encode(JSON.stringify(value)).length + key.length
I use the following code to check actual usage:
chrome.storage.<storage-area>.set({ [key]: value }, () => {
chrome.storage.<storage-area>.getBytesInUse(key, bytes => {
console.log("actual bytes in use", bytes);
});
});
Given a key of "test" and a value of "abc", the expected byte usage is 9b. The actual byte usage is 9b.
Given a key of "test" and a value of "«ταБЬℓσ»", the expected byte usage is 23b. The actual byte usage is 23b.
Given a key of "test" and a value of "<", the expected byte usage is 7b. The actual byte usage is 12b.
The storage is of course cleared between each check.
In the last example, what is causing those 5 extra, unexpected, bytes? What am I missing?
Edit: I'm using Google Chrome version 73.0.3683.75 (Official Build) (64-bit)
I found the reason thanks to w0xx0m's comment.
Chrome/Chromium replaces the less than character with "\u003C" to prevent script execution.
Source code can be found here.
Related
I am new to dynamo db binary data. I have a hash key + range key(both are byte[]). Now I am trying to get a list of items by querying on range key(ex: le, ge or between). I am able to do put and get operations fine.
However I am getting errors while doing this. My question is can dynamodb do this comparison? I am passing a byte[]. Can dynamodb check if existing rangekey(byte[]) is lesser or greater than this?
Yes, DynamoDB does support the byte array type well, and also allows comparison between them in conditions, done lexicographically, so what you want to do should and does work.
You didn't say which "errors" you are getting. You should be aware that DynamoDB treats the bytes of the byte arrays as unsigned bytes. For example, the byte 128 comes after byte 127. I don't know which language you are using to test this, but some languages have signed bytes - meaning that the byte 128 is treated as "-1" and will come before, not after, byte 127 in the sort order. DynamoDB doesn't do that, because it uses unsigned bytes.
I have a stream with ANSI string. It is prefixed with bytes length. How can I read it into std::string?
Something like:
short len = reader.readInt16();
char[] result = reader.readBytes(len); // ???
std::string str = std::copy(result, result + len);
but there is no method readBytes(int).
Side question: is it slow to read with readByte() from DataReader one byte at a time?
According to MSDN, DataReader::ReadBytes exists and is what you are looking for: http://msdn.microsoft.com/en-us/library/windows/apps/windows.storage.streams.datareader.readbytes
It takes an Platform::Array<unsigned char> as an argument, which presumably you'll initialize using the prefixed length, which on returning will contain your bytes. From there it's a tedious-but-straightforward process to construct the desired std::string.
The basic usage will look something like this (apologies, on a Mac at the moment, so precise syntax might be a little off):
auto len = reader->ReadInt16();
auto data = ref new Platform::Array<uint8>(len);
reader->ReadBytes(data);
// now data has the bytes you need, and you can make a string with it
Note that the above code is not production-ready - it's definitely possible that reader does not have enough data buffered, and so you'll need to reader.LoadAsync(len) and create a continuation to process the data when it is available. Despite that, hopefully this is enough to get you going.
EDIT:
Just noticed your side question. The short answer is, yes, it is much slower to read a byte at a time, for the reason that it is much more work.
The long answer: Consider what goes in to each byte:
A function call happens - stack frame allocation
Some logic of reading a byte from the buffer happens
The function returns - stack frame is popped, result is pushed, control returns
You take the byte, and push it into a std::string, occasionally causing dynamic re-allocation (unless you've already str.resize(len), that is)
Of all the things that happen, the dynamic reallocation is the really performance killer. That being said, if you have lots of bytes the work of function-calling will dominate the work of reading a byte.
Now, consider what happens when you read all the bytes at once:
A function call happens - stack frame, push the result array
(in the happy path where all requested data is there) memcpy from the internal buffer to your pre-allocated array
return
memcpy into the string
This is of course quite a bit faster - your allocations are constant with respect to the number of bytes read, as are the number of function calls.
I know that General GA data is sent by 1 pixel .gif request data. Where custom value [key, value] pair combined length had maximum fixed size. One limiting factor was IE older browsers that had maximum byte (not sure, but I assumed it was 2048 byte's for all values).
I assumed also that it was encoded in UTF-8 (variable-length multibyte encoding) where character can take 1-4 bytes in length. At some point in time I read that maximum character limit for custom value is 150 characters. At the time I did not need to know any better.
In case Universal GA the key part is stored in GA server and never sent. If I am not mistaken, each custom value has a separate request.
ga('create', 'UA-xxxxxxxx-x', 'xxx');
ga('set', {
'dimension1': '...',
'dimension2': '...',
'dimension3': '...',
'dimension4': '...',
'dimension5': '...'
});
ga('send', 'pageview');
My question is how meany bytes/characters '...' can have in case of Universal GA?
Encoded custom dimension value may not exceed 150 Bytes.
Best case: 150 characters
Worst case : 37 characters
Reference: link
I have 64000 small images I want to upload to my website (using existing validation, so no FTP etc). I've created an HTML5 [multiple] type=file input for this a while back to be used for a hundred or hundreds of images. Hundreds is not a problem. The images are batched and sent to the server.
But when I select a folder of ~ 16000 images, the file input's FileList is empty... The onchange event triggers, but the file list is empty. The browser (or file system or OS?) seems to have a problem selecting this many files.
I've created a very small tool to help determine what could be the max: http://jsfiddle.net/rudiedirkx/Ehhk5/1/show/
$inp.onchange = function(e) {
var l = 0, b = 0;
for (var i=0, F=this.files, L=F.length; i<L; i++) {
l += F[i].name.length;
b += F[i].size;
}
$nf.innerHTML += this.files.length + ' files: ' + (b/1000/1000) + ' MB / ' + l + ' chars of filename<br>';
};
All it does is count:
the number of files
the number of characters all file names are combined
the number of MB of total file size
When I try this, I get as very most:
1272 files: 176.053987 MB / 31469 chars of filename
(On 32 & 64 bit Win7, Chrome 26-52)
The next image (which fails) would be:
1273 images, which is not an obvious cut-off
between 176 and 177 MB filesize, also not an obvious cut-off
less than 32000 chars of filenames, also not an obvious cut-off, although it sort-of maybe looks like 32k...
In my calc, 1 MB = 1000^2 Bytes, not 1024^2. (That would be a MiB, but maybe my OS/filesystem/browser disagrees.)
My question would be: why this many files? Why this max? Is it OS dependent or browser dependent? Where do I find the specs for that? Is it JS' fault? Search for "file input max files" et al only results into the [max] attribute, which is irrelevant.
More test results:
In Firefox the max seems to be much higher. At least "2343 files: 310.66553999999996 MB / 60748 chars of filename" (that's all the files I have right here)
In Firefox also: "16686 files: 55.144415 MB / 146224 chars of filename" (much smaller, but more files)
Update
Chrome 52 canary Windows is still 32k of file name
Firefox (44+) Windows is still unlimited
why this many files?
The number of files depends on the number of characters all file names are combined.
Why this max?
In the Windows API, the maximum path length limitation is 256 chars, the Unicode version API is 32,767 chars.
Chrome simply sets the max path length of the Unicode version API, so it's about 32k chars as you observed.
Check this fix: https://code.google.com/p/chromium/issues/detail?id=44068
Firefox dynamically allocates a buffer big enough to hold the size of multiple selected files, which could handle much larger path length.
Check this fix: https://bugzilla.mozilla.org/show_bug.cgi?id=660833
Is it OS dependent or browser dependent?
Both.
Where do I find the specs for that?
For Windows API usage and reference:
http://msdn.microsoft.com/en-us/library/aa365247.aspx (Maximum Path Length Limitation)
http://msdn.microsoft.com/en-us/library/ms646839(VS.85).aspx
Is it JS' fault?
No.
I am pulling data from a Tektronix oscilloscope in Tektronix' RIBinary format using a TCL script, and then within the script I need to convert that to a decimal value.
I have done very little with binary conversions in the first place, but to add to my frustration the documentation on this binary format is also very vague in my opinion. Anyway, here's my current code:
proc ::Scope::CaptureWaveform {VisaAlias Channel} {
# Apply scope settings
::VISA::Write $VisaAlias "*WAI"
::VISA::Write $VisaAlias "DATa:STARt 1"
::VISA::Write $VisaAlias "DATa:STOP 4000"
::VISA::Write $VisaAlias "DATa:ENCdg RIBinary"
::VISA::Write $VisaAlias "DATa:SOUrce $Channel"
# Download waveform
set RIBinaryWaveform [::VISA::Query $VisaAlias "CURVe?"]
# Parse out leading label from scope output
set RIBinaryWaveform [string range $RIBinaryWaveform 11 end]
# Convert binary data to a binary string usable by TCL
binary scan $RIBinaryWaveform "I*" TCLBinaryWaveform
set TCLBinaryWaveform
# Convert binary data to list
}
Now, this code pulls the following data from the machine:
-1064723993 -486674282 50109321 -6337556 70678 8459972 143470359 1046714383 1082560884 1042711231 1074910212 1057300801 1061457453 1079313832 1066305613 1059935120 1068139252 1066053580 1065228329 1062213553
And this is what the machine pulls when I just take regular ASCII data (i.e. what the above data should look like after the conversion):
-1064723968 -486674272 50109320 -6337556 70678 8459972 143470352 1046714368 1082560896 1042711232 1074910208 1057300800 1061457472 1079313792 1066305600 1059935104 1068139264 1066053568 1065228352 1062213568
Finally, here is a reference to the RIBinary specification from Tektronix since I don't think it is a standard data type:
http://www.tek.com/support/faqs/how-binary-data-represented-tektronix-oscilloscopes
I've been looking for a while now on the Tektronix website for more information on converting the data and the above URL is all I've been able to find, but I'll comment or edit this post if I find any more information that might be useful.
Updates
Answers don't necessarily have to be in TCL. If anyone can help me logically work through this on a high level I can hash out the TCL details (this I think would be more helpful to others as well)
The reason I need to transfer the data in binary and then convert it afterwards is for the purpose of optimization. Due to this I can't have the device perform the conversion before the transfer as it will slow down the process.
I updated my code some and now my results are maddeningly close to the actual results. I assume it may have something to do with the commas that are in the data originally.
Below are now examples of the raw data sent from the device without any of my parsing.
On suggestion from #kostix, I made a second script with code he gave me that I modified to fit my data set. It can be seen below, however the result are exactly the same as my above code.
ASCIi:
:CURVE -1064723968,-486674272,50109320,-6337556,70678,8459972,143470352,1046714368,1082560896,1042711232,1074910208,1057300800,1061457472,1079313792,1066305600,1059935104,1068139264,1066053568,1065228352,1062213568
RIBinary:
:CURVE #280ÀçâýðüÿKì
Note on RIBinary - ":CURVE #280" is all part of the header that I need to parse out, but the #280 part of it can vary depending on the data I'm collecting. Here's some more info from Tektronix on what the #280 means:
block is the waveform data in binary format. The waveform is formatted
as: # where is the number of y bytes. For
example, if = 500, then = 3. is the number of bytes to
transfer including checksum.
So, for my current data set x = 2 and yyy = 80. I am just really unfamiliar with converting binary data, so I'm not sure what to do programmatically to deal with the block format.
On suggestion from #kostix I made a second script with code he gave me that I modified to fit my data set:
set RIBinaryWaveform [::VISA::Query ${VisaAlias} "CURVe?"]
binary scan $RIBinaryWaveform a8a curv nbytes
encoding convertfrom ascii ${curv}
scan $nbytes %u n
set n
set headerlen [expr {$n + 9}]
binary scan $RIBinaryWaveform #9a$n nbytes
scan $nbytes %u n
set n
set numints [expr {$n / 4}]
binary scan $RIBinaryWaveform #${headerlen}I${numints} data
set data
The output of this code is the same as the code I provided above.
According to the documentation you link to, RIBinary is signed big-endian. Thus, you convert the binary data to integers with binary scan $data "I*" someVar (I* means “as many big-endian 4-byte integers as you can”). You use the same conversion with RPBinary (if you've got that) but you then need to chop each value to the positive 32-bit integer range by doing & 0xFFFFFFFF (assuming at least Tcl 8.5). For FPBinary, use R* (requires 8.5). SRIBinary, SRPBinary and SFPBinary are the little-endian versions, for which you use lower-case format characters.
Getting conversions correct can take some experimentation.
I have no experience with this stuff but like googleing. Here are my findings.
This document, in the section titled "Formatted I/O Operations" tells that the viQueryf() standard C API function combines viPrintf() (writing to a device) with viScanf() (reading from a device), and examples include calls like viQueryf (io, ":CURV?\n", "%#b", &totalPoints, rdBuffer); (see the section «IEEE-488.2 Binary Data—"%b"»), where the third argument to the function specifies the desired format.
The VISA::Query procedure from your Tcl library pretty much resembles that viQueryf() in my eyes, so I'd expect it to accept the third (optional) argument which specifies the format you want the data to be in.
If there's nothing like it, let's look at your ASCII data. Your FAQ entry and the document I found both specify that the opaque data might come in the form of a series of integers of different size and endianness. The "RIBinary" format states it should be big-endian signed integers.
The binary scan Tcl command is able to scan 16-bit and 32-bit big-endian integers from a byte stream — use the S* and I* formats, correspondingly.
Your ASCII data clearly looks like 32-bit integers, so I'd try scanning using I*.
Also see this doc — it appears to have much in common with the PDF guide I linked above, but might be handy anyway.
TL;DR
Try studying your API to find a way to explicitly tell the device the data format you want. This might produce a more robust solution in the case the device might be somehow reconfigured externally to change its default data format effectively pulling the rug under the feet of your code which relies on certain (guessed) default.
Try interpreting the data as outlined above and see if the interpretation looks sensible.
P.S.
This might mean nothing at all, but I failed to find any example which has "e" between the "CURV" and the "?" in the calls to viQueryf().
Update (2013-01-17, in light of the new discoveries about the data format): to binary scan the data of varying types, you might employ two techniques:
binary scan accepts as many specifiers in a row, you like; they're are processed from left to right as binary scan reads the supplied data.
You can do multiple runs of binary scanning over a chunk of your binary data either by cutting pieces of this chunk (string manipulation Tcl commands understand they're operating on a byte array and behave accordingly) or use the #offset term in the binary scan format string to make it start scanning from the specified offset.
Another technique worth employing here is that you'd better first train yourself on a toy example. This is best done in an interactive Tcl shell — tkcon is a best bet but plain tclsh is also OK, especially if called via rlwrap (POSIX systems only).
For instance, you could create a fake data for yourself like this:
% set b [encoding convertto ascii ":CURVE #224"]
:CURVE #224
% append b [binary format S* [list 0 1 2 -3 4 -5 6 7 -8 9 10 -11]]
:CURVE #224............
Here we first created a byte array containing the header and then created another byte array containing twelve 16-bit integers packed MSB first, and then appended it to the first array essentially creating a data block our device is supposed to return (well, there's less integers than the device returns). encoding convertto takes the name of a character encoding and a string and produces a binary array of that string converted to the specified encoding. binary format is told to consume a list of arbitrary size (* in the format list) and interpret it as a list of 16-bit integers to be packed in the big-endian format — the S format character.
Now we can scan it back like this:
% binary scan $b a8a curv nbytes
2
% encoding convertfrom ascii $curv
:CURVE #
% scan $nbytes %u n
1
% set n
2
% set headerlen [expr {$n + 9}]
11
% binary scan $b #9a$n nbytes
1
% scan $nbytes %u n
1
% set n
24
% set numints [expr {$n / 2}]
12
% binary scan $b #${headerlen}S${numints} data
1
% set data
0 1 2 -3 4 -5 6 7 -8 9 10 -11
Here we proceeded like this:
Interpret the header:
Read the first eight bytes of the data as ASCII characters (a8) — this should read our :CURVE # prefix. We convert the header prefix from the packed ASCII form to the Tcl's internal string encoding using encoding convertfrom.
Read the next byte (a) which is then interpreted as the length, in bytes, of the next field, using the scan command.
We then calculate the length of the header read so far to use it later. This values is saved to the "headerlen" variable. The length of the header amounts to the 9 fixed bytes plus variable-number of bytes (2 in our case) specifying the length of the following data.
Read the next field which will be interpreted as the "number of data bytes" value.
To do this, we offset the scanner by 9 (the length of ":CURVE #2") and read so many ASCII bytes as obtained on the previous step, so we use #9a$n for the format: $n is just obtaining the value of a variable named "n", and it will be 2 in our case. Then we scan the obtained value and finally get the number of the following raw data.
Since we will read 16-bit integers, not bytes, we divide this number by 2 and store the result to the "numints" variable.
Read the data. To do this, we have to offset the scanner by the length of the header. We use #${headerlen}S${numints} for the format string. Tcl expands those ${varname} before passing the string to the binary scan so the actual string in our case will be #11S12 which means "offset by 11 bytes then scan 12 16-bit big-endian integers".
binary scan puts a list of integers to the variable which name is passed, so no additional decoding of those integers is needed.
Note that in the real program you should probably do certain sanity checks:
* After the first step check that the static part of the header is really ":CURVE #".
* Check the return value of binary scan and scan after each invocation and check it equals to the number of variables passed to the command (which means the command was able to parse the data).
One more insight. The manual you cited says:
is the number of bytes to transfer including checksum.
so it's quite possible that not all of those data bytes represent measures, but some of them represent the checksum. I don't know what format (and hence length) and algorithm and position of this checksum is. But if the data does indeed include a checksum, you can't interpret it all using S*. Instead, you will probably take another approach:
Extract the measurement data using string range and save it to a variable.
binary scan the checksum field.
Calculate the checksum on the data obtained on the first step, verify it.
Use binary scan on the extracted data to get back your measurements.
Checksumming procedures are available in tcllib.
# Download waveform
set RIBinaryWaveform [::VISA::Query ${VisaAlias} "CURVe?"]
# Extract block format data
set ResultCount [expr [string range ${RIBinaryWaveform} 2 [expr [string index${RIBinaryWaveform} 1] + 1]] / 4]
# Parse out leading label from Tektronics block format
set RIBinaryWaveform [string range ${RIBinaryWaveform} [expr [string index ${RIBinaryWaveform} 1] + 2] end]
# Convert binary data to integer values
binary scan ${RIBinaryWaveform} "I${ResultCount}" Waveform
set Waveform
Okay, the code above does the magic trick. This is very similar to all the things discussed on this page, but I figured I needed to clear up the confusion about the numbers from the binary conversion being different from the numbers received in ASCII.
After troubleshooting with a Tektronix application specialist we discovered that the data I had been receiving after the binary conversion (the numbers that were off by a few digits) were actually the true values captured by the scope.
The reason the ASCII values are wrong is a result of the binary-to-ASCII conversion done by the instrument and then the incorrect values are then passed by the scope to TCL.
So, we had it right a few days ago. The instrument was just throwing me for a loop.