In attempting to understand the concept of binary, my question is "How does a stored image or video look in binary on the hard drive?"
As for how it is physically stored, it depends on the technology of your storage device. For a hard disk drive you can read about it on Wikipedia.
The next layer is how the controller on the storage device sends the data to the motherboard.
Then how the motherboard sends the data to the operating system.
Then how the operating system stores the data on the disk (what file system it uses; NTFS is common in modern Windows installations.)
Finally, what you'll see when reading the data is groups of 8 bits (bytes) which are basically 8 on/off flags, which together form 256 possible combinations. Which is why most image formats are stored with colors varying from 0-255 for each channel (red, green, blue.) Most raw formats are stored linearly, so you can actually try reading them yourself. A raw image where the first pixel is red (assuming it stores the pixels left-to-right, top-to-bottom) would look like this in bits:
11111111 00000000 00000000
red green blue
For more information, you'll have to be more specific.
Every file on disk is basically a number of bits in a row.
The difference between "binary" and "something else" (often called ASCII, or text, or...) is that non-binary is basically human readable when opened in a text editor. In other words: the bytes in the file map to human readable letter (and other) characters in some way a generic text editor knows how to handle.
So called binary files can only be interpreted back to that data that they actually contain when you know the format which was used to map the content (image, sound, movie, whatever) to a stream of zeros and ones. This mapping is called the file format and is usually part of the file name in the form of an extension. You need a piece of software that knows the mapping and can interpret the row of bits back into the original content.
Mind you: this is usually only a hint. Renaming a JPEG image file to have a .mp3 extension doesn't change it into an audio file; it is still just an image file, containing the image (=dimensions of the image in pixels + the color values for each pixel, basically) encoded into a stream of zeros and ones in the way described in the JPEG file format encoding description.
Check out the link: Binary File Format
The images are sequential flow of colored dots... But it's not hardware dependent i.e. your hard-disk will store any thing in any format which your OS provide it to... However the OS maintain standards of saving file formats other wise a JPG image will not be valid one across different platforms...
Simillarly the videos are flows of images and voice data multiplexed into a sequential flow.
All data on commercial computer systems are stored in binary format (we'll ignore scientific studies into quantum and optical computing).
At the lowest level all files and processing by a computer are performed in binary. This is because our computing systems are powered by the flow of electrons. They either flow or don't. Electric current is on or off. 1 and 0.
The data stored on a hard disk is there due to pulsing of the hard disk write head coil which magnetises spots of hard disk material. These magnetised spots cause a current pulse in the read coil (in actual fact the read and write coils are the same) as the hard disk head passes over them. Hence the data is read as a stream of current pulses, 1s and 0s.
Now processors are built to accept process a finite number of binary "pulses" or data bits simultaneously (it can be anything from 4 bits upwards). Hence a modern 64bit PC can process 64 binary data bits i.e. 64 1s and 0s, at any one time.
Now at a higher level, although all files are stored as binary and can be read in binary format we help the processing of them by telling the processor what format to read them in. This is so that it process the file data as small chunks e.g. 8 bits or 1 byte for ASCII text.
The operating system provides the processor with a template for any given file. This is set up in an extension relation table. And according to what the file extension is the operating system will expect that data to be in a particular format and link it to code that can be used by the processor to interpret it. Hence changing a file name extension will confuse the processor as it won't interpret the data correctly. That's why changing the filename from *.jpg to *.exe won't show the image, as the processor has been told to expect executable code, which the data within the file clearly isn't.
So back to your original question the image within the jpeg file has been encoded as series of 1s and 0s in a specific order.
I'm not sure how exactly they are arranged, but as an example:
A picture was captured and stored as a bitmap at a resoultion of 800 x 600 in 24bit colour. The first pixel is stored as 3 bytes (8 bit binary) representing a red, green and blue value. The value of each byte dictates the intensity of that colour. 0 - 255, with 0 being none at all to 255 being the highest value. Unsigned 255 in binary is 11111111, I won't confuse you with 2's complement for signed values. So the full picture will require a file of minimum 1,440,000 bytes or about 1,406 kilobytes (a kilobyte being 1024 bytes).
The binary as follows: 000010101011010101101010101 would be stored on a hard drive actual microscopic bumps and troughs by changing the polarity of the metalic grains on the disk in specific regions.Binary is actually read from right to left, obviously the opposite way of how most people read text.
If your question is really "how does it look": See Figure 4 on this page; it shows high resolution measurements of a hard drive.
Although googletorp's answer does not look very helpful, it's not totally untrue. To store binary data, the only thing you need is the possibility to have two different states for each storage unit (be it an on/off switch, hole or no hole in a punchcard, or, as in the case of hard drives, the direction of ferromagnetic particles).
The Wikipedia page for the BMP File Format contains an example(Including all hex values) of a 2x2 pixel bitmap image, it should be very good at explaining the basis of the binary representation of an image.
In general if you're really curious how the binary looks for a file you could always use a Hex Viewer and take a look yourself :) I normally use od on Linux to dump the binary information of a file. I'm sure you can google a good Hex Editor for Windows (or maybe someone can suggest one.)
Headers ? Every file created contains header information, that are also stored as binary bits along with the data. The header bits of a files holds the information of header length, file type, file location and length. Now each application is designed to read certain file types. If the application tries to open a file on hard disk which has a header with a different file format, that is not supported by the application, it fails to read the file. Thus a text file cannot be opened using a media player. Because a media player expects a file that contains a header with audio file format binary pattern. Similarly, same in case of picture files.
Related
I have an Octave code that gathers data from thousands of .csv files and stores it in a 4-dimensional matrix (800x8x80x213) so I can access it with other code. The process of reading in the data takes about 10 minutes so I thought it would be a good idea to save the matrix and then I could load it into the workspace whenever I wanted to work with the data instead of waiting 10 minutes for the matrix to be created. I used Save to save the matrix and Load to load it into the workspace, however when I loaded the matrix, it took 30 minutes to complete. Is there a better/faster way to save/load this 4-D matrix? Seems ridiculous that it takes 3 times longer to load a matrix than to create it from 4000+ files...
The default 'format' option used by the save command is -text, which is human readable. For large datasets, this will take a long time to create (not to mention, it will lead to a much larger file, since it will need to represent floating point numbers via their text representations...), so it is indeed inappropriate for this kind of data. Loading from a large text format file will also take quite a long time, especially on a slow computer, for the same reasons.
Octave also supports a -binary option, which is octave's internal binary format. This is what you need. E.g.
save -binary outputfile.bin varname
In this particular case, the text file is 2.2G, whereas the binary format is the expected 872Mb (i.e. number of elements * 8 bytes per element). Saving and loading is near instant.
Alternatively, there's a bunch of other options too, corresponding to other common formats, e.g. as a commenter has also mentioned here, -hdf5, or -v7 which is matlab's .mat format.
Type help save on your octave console for more details.
I'm working on storing a JSON snippet onto a type 31 QR Code so that I can scan it with a smartphone and parse the JSON.
I'm running into a few challenges..
A type 31 QR Code is the "densest" (for lack of better words) code that I can get my Android device to reliably scan. This can store 2677 Alphanumeric characters factoring in 7% error correction.
What are my options for compressing my optimized/minified JSON object and encoding a QR code with it? Conceivably, how much more data can I store? Or am I even barking up the right tree?
It all depends, really.
Is Wi-Fi available? If so, put your JSON snippets on a web server and encode their URLs in the QR codes. Problem solved.
If this is for general consumption, then you need to be aware that some phones are better than others. Mine really struggled to scan a version 25 QR code. I'd consider anything higher than version 20 as unreliable.
There's little benefit in using the alphanumeric mode. It only stores uppercase letters, digits 0-9 and a handful of punctuation marks. At 5½ bits per character (11 bits per pair), its storage capacity is almost identical to the corresponding binary mode (8 bits per character).
In a quick test gzip -n -9 reduced a 545-byte JSON file to 219 bytes (40% of the original size). You could do a lot better than this if you stored your data in a compact binary format instead of a verbose tagged format.
If you're putting these QR codes out in public, you'll need to include some sort of authentication mechanism (e.g., a 32-bit checksum) to prevent malicious code injections and other tomfoolery.
So the question is: How does a computer go from binary code representing the letter "g" to the correct combination of pixel illuminations?
Here is what I have managed to figure out so far. I understand how the CPU takes the input generated by the keyboard and stores it in the RAM, and then retrieves it to do operations on using an instruction set. I also understand how it does these operations in detail. Then the CPU transmits the output of an operation which for this example is an instruction set that retrieves the "g" from the memory address and sends it to the monitor output.
Now my question is does the CPU convert the letter "g" to a bitmap directly or does it use a GPU that is either built-in or separate, OR does the monitor itself handle the conversion?
Also, is it possible to write your own code that interprets the binary and formats it for display?
In most systems the CPU doesn't speak with the monitor directly; it sends commands to a graphics card which in turn generates an electric signal that the monitor translates into a picture on the screen. There are many steps in this process and the processing model is system dependent.
From the software perspective, communication with the graphics card is made through a graphics card driver that translates your program's and the operating system's requests into something that the hardware on the card can understand.
There are different kinds of drivers; the simplest to explain is a text mode driver. In text mode the screen is composed of a number of cells, each of which can hold exactly one of predefined characters. The driver includes a predefined bit map font that describes how a character looks like by specifying which pixels are on and which are off . When a program requests a character to be printed on the screen, the driver looks it up in the font and tells the card to change the electric signal it's sending to the monitor so that the pixels on the screen reflect what's in the font.
The text mode has limited use though. You get only one choice of font, a limited choice of colors, and you can't draw graphics like lines or circles: you're limited to characters. For high quality graphics output a different driver is used. Graphics cards typically include a memory buffer that contains the contents of the screen in a well defined format, like "n bits per pixel, m pixels per row, .." To draw something on the screen you just have to write to this memory buffer. In order to do that the driver maps the buffer into the computer memory so that the operating system and programs can use the buffer as if it was a part of RAM. Programs then can directly put the pixels they want to show, and to put the letter g on the screen it's up to the application programmer to output pixels in a way that resembles that letter. Of course there are many libraries to help programmers do this, otherwise the current state of the graphical user interface would be even sorrier than it is.
Of course, this is a simplification of what actually goes on in a computer, and there are systems that don't work exactly like this, for example some CPUs do have an integrated graphics card, and some output devices are not based on drawing pixels but plotting lines, but I hope this clears the confusion a little.
See here http://en.m.wikipedia.org/wiki/Code_page_437
It describes the character based mechanism used to display characters on a VGA monitor in character mode.
I have a simple question.
When we create a file, let's say "abc.txt" and leave it blank.
The OS will show that the file is 0 bytes size and takes 0 bytes on disk.
If we save 100 of these 0 bytes file into a folder, the OS will also say that the folder's total size is 0 bytes.
This may sound logical because there is nothing in the file. But should not these files take at least a few bytes in the storage device?
After all, we save it somewhere and named it something. Shouldn't the file's name and possibly some other headers at least takes up some space?
No, they still occupy a few bytes on the file system. Otherwise I would implement a magic filesystem that stored everything encoded in the filenames on empty files.
It actually boils down to a matter of definition. Either the "size of a file" refers to the size of the content of the file, or it refers to the "difference" it makes in terms of free bytes on the underlying file system (that is, size of content (rounded up to the closest block- or cluster-size) + bytes used for it's inode).
These details are stored into what is known as File Allocation Table (talking in Windows FAT context) traditionally. They are created when we format the hard drive. Some predefined space is allocated for it. I don't think the size of it changes.
For example, you format a 100 GB hard drive, only 90+ GB is available for you to use. Other space is used by the file system to manage/remember each file/folder that is saved on the hard drive and where it is saved.
The answer to this question is file-system dependent.
For example, on NTFS an empty file takes up a cluster, and a cluster has a size that depends on your hard disk size.
Here you can read some common cluster size for Windows' file systems.
The fact that they are present on disk means that a record has been created for them, which of course requires some amount of memory. The 0 bytes simply corresponds to the logical size of the file rounded to the granularity displayed in the UI, but even then it likely contains a file header which will depend on the file format.
I'm trying to decide if I should generate AS3 code for a certain complicated configurable set of business-logic rules, or should I express them as data, and write a state machine in AS3 to interpret it.
My goal is to get smallest compiled swf size. Speed is not an issue. Implementation complexity is not an issue as well. (Both within rational limits, of course.)
I can not disclose sufficient details, and I understand that I probably should do experiment instead of asking, but my question is:
What is average compression ratio for AS3 source when compiled to swf? How much bytes of swf per kilobyte of source code?
(I perfectly understand that the answer would be a very rough figure at best.)
Some facts about the compiled SWF:
class names are preserved
member names are preserved
local variable names are not preserved
comments and whitespace are stripped
So, the ratio would depend on length of identifiers, comments and even tabs-or-spaces preferences. If you run your result SWF through obfuscator which replaces class names with something like A0, A1, etc. you should save some bytes.
Your state machine idea seems to be the most promising - code is written once, and rules can be written in compact way. If you can pack small numbers into one int, it will be better (there are no integers in AS3 less than 4-byte ints and uints).
Remember that SWF files are already zlib compressed so that for the most part might erase any relative gains by going either data way or code way.
If the initial load speed is what you are going after you can always factor out chunks of your code and load it on demand from another SWF file.