Is it possible to find IPFS hashes by their leading bits? Lets say I have some files and by adding a nonce to each of them I create hashes which all start with the same n Bits. Is it possible for a client to find these files by searching for hashes that start with these bits?
If so, how is it done?
TLDR: No
The two main ways IPFS looks for data that is available on the network are:
Using a Distributed Hash Table which allows asking peers "do you know who has the data with hash X?"
Asking peers you are already connected to "do you have the data with hash X?"
This means that there is no efficient way to ask "do you know who has the data with hash starting with the bits Y?".
Related
In my project, I need to download data from ipfs by giving a CID.
What I do is:
ipfs pin add {CID}
ipfs get {CID}
But I found these two steps are quite time-consuming, it takes at least 1min above.
I tried localhost and infura.
What can I do to let it download faster?
When you just want to download files, there is no need to pin them first. This might save a (tiny) bit of overhead.
However, the bulk time is probably spent in
Looking up nodes in the distributed hash table that provide your data, and
Actually transferring the data from these nodes.
For small data sizes, the first item is probably the limiting factor. In general, the duration depends on the number of nodes that host your connects to and how fast these nodes (can) transfer the data to your client.
How to locate data in distributed network which is based on IPFS protocol if I don't know the hash value of that data.
I am new to IPFS. I know a little bit about IPFS and how it works. I came to know that IPFS is protocol which is content-addressed and user can retrieve the data by specifying the hash value of that data.
In a distributed network how will the 2nd user will be knowing the hash value of some data that 1st user added to the IPFS.
I went through some resources and IPFS site and came to know that it works on Distributed Hash Tables. But I am still not clear.
Please help me with the topic.
This question already has answers here:
Storing Images in DB - Yea or Nay?
(56 answers)
Closed 9 years ago.
I have a site with more than 100k static files in a single directory (600k+ dirs and files in total). I guess I could get a VPS to host it without inode issues, but it won't be a high traffic site, so I'd rather use a cheap webhost.
I'm thinking to store the files in a MySQL table indexed by URL path and serve through PHP. Are there better approaches?
EDIT: Just to clarify, this is NOT the same as storing images on the DB. I'm talking about HTML pages.
I think your best approach would not be to store them in the database to start with. When it comes to storing and serving files, that is what a file system does best. There are no possible reasons that a database can do this more efficiently that a normal file system.
If you were to store them in a database then given the size restrictions you would want to use a BLOB field (e.g. TEXT) and for efficiency hash the URL and store that in a column rather than having some huge VARCHAR field indexed.
However, as you've said they are static there really isn’t any point in this – as they are static have your webserver add some long caching headers to the pages so they will be stored locally for future hits from the same client.
[Edit 1 - in response to comment]
I was answering the question with the information given and keeping it generic where information wasn't provided by OP.
It depends on how much of the VARCHAR you index – which is related to the length of the data stored (URL / path / page name) you’re indexing.
If you’re indexing less than about 45 characters for only 100k rows I guess it really wouldn't make much difference, a hash will use less memory but size and performance for a small set probably wouldn't really make that much difference.
I answered it as the OP asked about the database but still can’t see any reason why you would want to put them there in the first place – it will be slower than using the file system.0 Why connect to the database, deal with network performance (unless they are on the same box – unlikely in a web host) query an index, fetch a row, run that data through the database provider and stream the output to the response stream when the webserver can do the same outcome with much less CPU cycles and in comparison to a database a fraction of the memory usage?
Yes - a filesystem is a database. All the filesystems I've come across in the last 10 years can easily accommodate this number of files in a directory - and the directories are implemented as trees (there are some using B-Trees - but structures with bigger fanouts such as H-Trees work better for this kind of application).
(actually, given the coice I'd recommend structuring it into a hierarchy of directory - e.g. using dirs for the first 2 letters of the filename or md5 hash of the content - it'd make managing the content a lot easier without compromising performance).
Relational databases are all about storing small pieces of structured data - they are not an efficient way to manage large variable sized data.
I don't have any benchmarks to hand but just as I'd pick a station wagon to move several petabytes of data quickly over a sports motorcycle, I'd go with a suitable filesystem (such as BTRFS or Ext4 - ZFS would do the job too but it's not a good choice on anything other than Solaris - and it's questionable whether solaris makes any sense for a webserver).
Problem is that cheap hosting companies rarely provide this level of information up front.
Note that a wee tweak of the filesystem behaviour can yield big imperovements in performance - in your case, if running on Linux, I'd recommend reducing the vfs_cache_pressure significantly. But this requires root access.
An alternative approach would be to use a document database rather than a relational database (not a key/value store). These are a type of Schema free (NoSQL) database designed to provide fast replication and handling of large datastructures. Hence this would provide a more scalable solution (if that's a concern). e.g. RavenDB. You could use a key/value store but these are rarely optimized to handle large data payloads.
I'd only consider MySQL if you have a very strong reason other than what you've described here.
Long time reader, first time poster. And I start with quite a cryptic one!
What I'm seeking to do is encrypt a string with the SHA-256 algorithm, and hash it with a key.
I discovered someone had done some excellent work in creating an algorithm for "normal" SHA-2 encryption as a stored function at: http://blog.darkrainfall.org/sha-256-in-mysql/ which will probably be of help to others, but I need to be able to do it with a key.
Anyone know if this is possible? I'm a completely newbie to encryption I'm afraid.
I'm using mySQL 5.1 on Windows 2003 server.
Cheers.
It is a little unclear what your end goal is, but the SHA implementation you referenced should be able to do the hashing you desired. One meaning of "hashing something with a key" for message authentication might be that you take a secret key and prepend it to data and then hash the entire result. The ever-useful Wikipedia has some information on HMAC.
Note that hashing is not encryption. The question seems to imply that hashing something is the same as encrypting it. A hash, though, takes some data and runs it through a data blender and produces a (typically) fixed length chunk of data. With a cryptographically strong hash function, it is supposed to be impossible (from a practical standpoint) to find an input that results in a given hash value. Encryption, on the other hand, takes a key and a chunk of data and runs i through a data blender and produces a chunk of data that can then be "unblended" in conjunction with the original key to produce the original data.
I have a web application that uses a symmetric encryption algorithm.
How would you store the secret key and initialization vector? Storing as a literal in the code seems like a bad idea. How about app settings? What is the best practice here?
One standard approach in the webapp world is to split the key and put it in different places. E.g., you might split the key and put part of it in the filesystem (outside of the 'webapps' directory), part of it in the JNDI configuration (or .net equivalent), and part of it in the database. Getting any single piece isn't particularly hard if you're compromised, e.g., examining backup media or SQL injection, but getting all of the pieces will require a lot more work.
You can split a key by XOR-ing it with random numbers of the same size. (Use a cryptographically strong random number generator!) You can repeat this process several times if you want to split the key into multiple pieces. At the end of the process you want, e.g., three partial keys such that p1 ^ p2 ^ p3 = key. You might need to base64-encode some of the partial keys so they can be stored properly, e.g., in a JNDI property.
(There are more sophisticated ways to split a key, e.g., an n-of-m algorithm where you don't require all of the pieces to recreate the key, but that's -far- beyond what you need here.)
If you can require the user to actively enter the password, there are PBE (password-based encryption) algorithms that convert a password to a good symmetric key. You want to find one that requires an external file as well. Again it's a case the tape backups or the password itself isn't enough, you need both. You could also use this to split the password into two pieces with JNDI - you can use a plaintext passphrase in JNDI and an initialization file somewhere in the filesystem.
Finally, whatever you do be sure you can 'rekey' your application fairly easily. One approach is to use the password obtained above to decrypt another file that contains the actual encryption key. This makes it easy to change the password if you think it's been compromised without requiring a massive reencryption of all of the data - just reencrypt your actual key.
Is it possible for you to enter a password interactively whenever the application starts up? That way you don't have to store the key, or at least any keys (whether they are symmetric or private keys) can be encrypted with this "bootstrap" password.
If not, store your secret key in a file by itself and modify its permissions to make it accessible only to the user running the web application.
These approaches are platform-agnostic. For more concrete suggestions, information about your platform would be helpful.
By the way, an initialization vector should be used for only one message. And IVs do not have be kept secret, so you could store it anywhere, but storing it with the one message that uses it is customary.
I have used an approach where my application requires a symmetric key when it starts and looks for it in a certain file. Once the application has started up I remove the file. A copy of the file is kept remotely for any required restarts. Obviously this approach is not viable if your applciation has frequent restarts.
Another alternative would be a certificate manager such as the Windows Certificate Store. It can store certificates and their keys securely and it's also possible to mark private keys as non-exportable so it would require some serious hacking to get the key out. Your application could load its certificate from the Certificate Store and be able to call operations to sign requests or generate new symmetric keys. In addition you can assign permissions to different certifcate stores so that only certain privileged accounts would be able to access the certificate.
stick it in the web.config and encrypt that section
This SO question talks more about web.config encryption
This should help ...
http://msdn.microsoft.com/en-us/library/ms998280.aspx
But, you really should consider going to PKI if you are serious about protecting your data.
We have a slightly different, but related issue. We have keys generated every few days, and when decrypting, we have to try all our keys because we do not know which day the encryption took place. What we did was to encrypt the keys one more time and store them as secretes. This way, we only have one set of keys to manage.
For secure storing of encryption key you can use KMS service of AWS. Please use this service for storing such confidential keys. PFB url for kms service.
documentation : https://aws.amazon.com/kms/