When I delete solr's index files on disk, (found in /solr/data/index and solr/data/spellchecker), solr throws an exception whenever I try to make a request to it:
java.lang.RuntimeException: java.io.FileNotFoundException: no segments* file found in org.apache.lucene.store.NIOFSDirectory#/…/solr/data/index:
The only way I've found to recover from this is to “seed” the data directory with the index files from elsewhere. It doesn't really matter where it seems. Once I do this, I can run a query to reload the schema and regenerate the index. Is this how this is supposed to work? It seems like there should be a way to tell solr to regenerate those files from scratch. Maybe I'm just mistaken in my assumption that these files are not part of the application itself (kind of implied by the name “data”)?
Solr will throw that exception at startup if the index directory exists but is empty. However if you delete the directory, Solr will create it and the empty segments files at startup.
If you are using sunspot solr on rails, sunspot can reindex all the data from the database into solr. However, solr standalone would not know where to pull the data to reindex. You would need a backup of the data.
The problem may lie with the segments file if you delete the index . the files are physically deleted but are present in the ram or cache of the solr . Avoid deleting files directly from solr index files physically . use delete query to delete the index , doing this would alter the segments of the index and you will not have to restart the solr
regards
Rajat
Exception FileNotFoundException signals that an attempt to open the file denoted by a specified pathname has failed. So either your index is invalid or corrupted.
NIOFSDirectory class is used for reading and writing index files. The directory is created at the named location if it does not yet exist.
So you should probably:
Delete the index directory or restore data from backups.
Restart the server (or at least the reload the config).
Related
I am working on designing a little project where I need to use Consul to manage application configuration in a dynamic way so that all my app machines can get the configuration at the same time without any inconsistency issue. We are using Consul already for service discovery purpose so I was reading more about it and it looks like they have a Key/Value store which I can use to manage my configurations.
All our configurations are json file so we make a zip file with all our json config files in it and store the reference from where you can download this zip file in a particular key in Consul Key/Value store. And all our app machines need to download this zip file from that reference (mentioned in a key in Consul) and store it on disk on each app machine. Now I need all app machines to switch to this new config at the same time approximately to avoid any inconsistency issue.
Let's say I have 10 app machines and all these 10 machines needs to download zip file which has all my configs and then switch to new configs at the same time atomically to avoid any inconsistency (since they are taking traffic). Below are the steps I came up with but I am confuse on how loading new files in memory along with switch to new configs will work:
All 10 machines are already up and running with default config files as of now which is also there on the disk.
Some outside process will update the key in my consul key/value store with latest zip file reference.
All the 10 machines have a watch on that key so once someone updates the value of the key, watch will be triggered and then all those 10 machines will download the zip file onto the disk and uncompress it to get all the config files.
(..)
(..)
(..)
Now this is where I am confuse on how remaining steps should work.
How apps should load these config files in memory and then switch all at same time?
Do I need to use leadership election with consul or anything else to achieve any of these things?
What will be the logic around this since all 10 apps are already running with default configs in memory (which is also stored on disk). Do we need two separate directories one with default and other for new configs and then work with these two directories?
Let's say if this is the node I have in Consul just a random design (could be wrong here) -
{"path":"path-to-new-config", "machines":"ip1:ip2:ip3:ip4:ip5:ip6:ip7:ip8:ip9:ip10", ...}
where path will have new zip file reference and machines could be a key here where I can have list of all machines so now I can put each machine ip address as soon as they have downloaded the file successfully in that key? And once machines key list has size of 10 then I can say we are ready to switch? If yes, then how can I atomically update machines key in that node? Maybe this logic is wrong here but I just wanted to throw out something. And also need to clean up all those machines list after switch since for the next config update I need to do similar exercise.
Can someone outline the logic on how can I efficiently manage configuration on all my app machines dynamically and also avoid inconsistency issue at the same time? Maybe I need one more node as status which can have details about each machine config, when it downloaded, when it switched and other details?
I can think of several possible solutions, depending on your scenario.
The simplest solution is not to store your config in memory and files at all, just store the config directly in the consul kv store. And I'm not talking about a single key that maps to the entire json (I'm assuming your json is big, otherwise you wouldn't zip it), but extracting smaller key/value sets from the json (this way you won't need to pull the whole thing every time you make a query to consul).
If you get the config directly from consul, your consistency guarantees match consul consistency guarantees. I'm guessing you're worried about performance if you lose your in-memory config, that's something you need to measure. If you can tolerate the performance loss, though, this will save you a lot of pain.
If performance is a problem here, a variation on this might be to use fsconsul. With this, you'll still extract your json into multiple key/value sets in consul, and then fsconsul will map that to files for your apps.
If that's off the table, then the question is how much inconsistencies are you willing to tolerate.
If you can stand a few seconds of inconsistencies, your best bet might be to put a TTL (time-to-live) on your in-memory config. You'll still have the watch on consul but you combine it with evicting your in-memory cache every few seconds, as a fallback in case the watch fails (or stalls) for some reason. This should give you a worst-case few seconds inconsistencies (depending on the value you set for your TTL), but normal case (I think) should be fast.
If that's not acceptable (does downloading the zip take a lot of time, maybe?), you can go down the route you mentioned. To update a value atomically you can use their cas (check-and-set) operation. It will give you an error if an update had happened between the time you sent the request and the time consul tried to apply it. Then you need to pull the list of machines, and apply your change again and retry (until it succeeds).
I don't see why you would need 2 directories, but maybe I'm misunderstanding the question: when your app starts, before you do anything else, you check if there's a new config and if there is you download it and load it to memory. So you shouldn't have a "default config" if you want to be consistent. After you downloaded the config on startup, you're up and alive. When your watch signals a key change you can download the config to directly override your old config. This is assuming you're running the watch triggered code on a single thread, so you're not going to be downloading the file multiple times in parallel. If the download failed, it's not like you're going to load the corrupt file to your memory. And if you crashed mid-download, then you'll download again on startup, so should be fine.
In the scenario where Logstash crashes or the Elasticsearch server is not reachable, I might have to restart Logstash and begin processing a file that was half-way inserted into Elasticsearch.
Does Logstash remember which line in the log file was last processed and pick up where it left off, or will it insert duplicate documents?
I suspect that the _id could be a hash generated by the file and line number to avoid duplicates but I am not sure.
The Elastic products that read files (logstash, filebeat, or the older logstash-forwarder) remember where they are in the files that they're reading.
If the pipeline backs up, each component will then stop sending more logs until the congestion is removed.
There will be logs "stuck" in the pipeline. IIRC, the logstash queue is 20 events. If you kill logstash before it can write those, you'll lose those events. They were working on making that better for logstash2, but it didn't make it in.
So, the risk is more of missing a few documents than getting duplicates.
I have a TokuDB table that for some reason has a missing ***_status.tokudb file.
I am not yet sure, whether the file is missing due to a TokuDB crash or not.
Question is:
Is there a way to recover or recreate the status file from the main and key files (which I can see are present from tokudb_file map.)??
How can I debug what caused the tokuDB status file to get deleted ?
Is this really frequent or a known bug ?
https://github.com/percona/tokudb-engine/wiki/Broken-tables-caused-by-non-transactional-table-operations#unexplained-inconsistency-problems-with-tokudb
So, I was able to recover my files from the main files.
I still don't know what deleted the status files though.
The toku-ft repository has an internal debugging tool called tokuftdump.
After it parses the tree, it dumps bytestreams on the leaf entries that are unpacked. Some quick hex editing on the converted hexstreams reveals the structure, and then you can modify the utility to dump the exact values post parsing as revealed by the structure.
Since toku has message buffers on nodes, you also may need some additional message processing. In my case this was simple since I only had inserts...
Update: More details can be found here.
http://kshitij.learnercafe.com/TokuDB-Recovery-From-Files
I am trying to save or retrieve data from my co-workers computer. Her MySQL instance does not work anymore (cannot figure out the reasons). I would like to know if there is a way to save all the data from the databases present in her computer.
Any suggestions on how to save the data, and maybe re-install MySQL again?
You can copy the tables to another computer with a working MySQL instance, as long as the floating point format is the same on both computers (see this page for reference, covered in the first paragraph).
As long as you copy the .frm, .myd and .myi files somewhere safe, you can also try reinstalling MySQL and see if that works for you.
First of all, errors may be recoverable. If you do not have ability to recover from errors, you can try to make a plain copy of MySQL data. BUT! If tables used InnoDB engine, that will be not so easy and chance of recovery will be not so big.
Recently, I write a python code to insert HTML text into table. After my code writing about 200,000 html page, I can use select to retrieval all these data. However, I find that the MySQL server does not write any data into files. I check the memory usage and find that mysqld.exe program consumes more than 1.5GB memory. I search the whole disk about the table name but I only found a 9KB file related to my table name. By the way, I also checked the mysql.ini file. The path configuration is correct. Then, I use mysqldump to backup that table. This command gives me more than 7GB sql file. I check it again and find there is 20GB file ibdata in my datadir folder. What is that file mean? Why there is no file related to my table ? Does MySQL just store the data in memory?
Run SHOW TABLE STATUS and check the storage engine value. It might be using MEMORY, though I would be surprised if it is and you didn't know that (because you would have to set it explicitly).
Maybe you're on InnoDB but haven't enabled innodb_file_per_table, so everything's being written to one file.