Is there a tool or script that would read my server properties (Number of CPUS, Cores, Memory layout and Structure) and automatically generate a full system mode (FSE) configuration script based on that ?
I know this might little ambitious, but maybe something like that exists somewhere ....
Related
I've read multiple times that I can cause read/write errors if I create a snapshot. Is it possible to create a snapshot of the disk my machine is booted off of?
It depends on what you mean by "snapshot".
A snapshot is not a backup, it is a way of temporarily capturing the state of a system so you can make changes test the results and revert back to the previously known good state if the changes cause issues.
How to take a snapshot varies depending on the OS you're using, whether you're talking about a physical system or a virtual system, what virtualization platform, you're using, what image types you're using for disks within a given virtualization platform etc. etc. etc.
Once you have a snapshot, then you can make a real backup from the snapshot. You'll want to make sure that if it's a database server that you've flushed everything to disk and then write lock it for the time it takes to make the snapshot (typically seconds). For other systems you'll similarly need to address things in a way that ensures that you have a consistent state.
If you want to make a complete backup of your system drive, directly rather than via a snapshot then you want to shut down and boot off an alternate boot device like a CD or an external drive.
If you don't do that, and try to directly back up a running system then you will be leaving yourself open to all manner of potential issues. It might work some of the time, but you won't know until you try and restore it.
If you can provide more details about the system in question, then you'll get more detailed answers.
As far as moving apps and data to different drives, data is easy provided you can shut down whatever is accessing the data. If it's a database, stop the database, move the data files, tell the database server where to find its files and start it up.
For applications, it depends. Often it doesn't matter and it's fine to leave it on the system disk. It comes down to how it's being installed.
It looks like that works a little differently. The first snapshot will create an entire copy of the disk and subsequent snapshots will act like ordinary snapshots. This means it might take a bit longer to do the first snapshot.
According to :
this you ideally want to shut down the system before taking a snapshot of your boot disk. If you can't do that for whatever reason, then you want to minimize the amount of writes hitting the disk and then take the snapshot. Assuming you're using a journaling filesystem (ext3, ext4, xfs etc.) it should be able to recover without issue.
You an use the GCE APIs. Use the Disks:insert API to create the Persistence disk. you have some code examples on how to start an instance using Python, but Google has libraries for other programming languages like Java, PHP and other
I want to be able to read from a real live proper MySQL database using standard file access routines. I don't mean reading the MySQL database's own underlying private files. What I mean is implementing a file-based linux device driver that "presents" a MySQL database as a file. In other words, the text file is a "View" of the MySQL database. The MySQL records are presented in our homegrown custom variation of the CSV format that the legacy code was originally written to understand.
Background
I have some legacy code that reads from a text file that contains a very large table of data, each line being a separate record. New records (lines) need to be added but there is contention for the file among the team, there is also an overhead in deployment of the legacy code and this file to many systems when releasing the software to them. The text file itself also needs to be version controlled.
Rather than modify the legacy code to call a MYSQL database version of these records directly, I thought it would be better to leave it untouched. This would avoid risks in modifying the code and ease deployment and moreover, modifying the code would cause much overhead in de-risking, design discussions, more testing etc.
So what I'm looking to do is write a file-based device driver such that this makes the MySQL database appear as a file to the legacy code, with the data within the format that the legacy code expects. That way the legacy code is not changed and can work oblivious that the file is really an underlying database. Contention is removed because the individual records in the database can now be updated/added to separately (via MySQL, or even better a separate web admin interface that guides and validates data entry from the user for individual records) and deployment effort is much reduced without having to up-issue the whole file on all the systems that use it.
The device driver would contain routines to internally translate standard file read operations into MySQL queries to the MySQL database and contain routines to return the MySQL results and translate these into the text format for returning back to the file read operation.
This is for a Linux/Unix platform.
Has this been done and what are your thoughts?
(cleaned up the question, grammar, clarification, readability. This does not affect the accepted answer.)
This kind of thing has been done before - an obvious example being the dynamic view filing system in ClearCase which provided (maybe still does?) a virtualised view onto a version control repository. Behind the scenes it implemented an object cache and used RPC to fetch objects from other hosts if necessary, and made extensive use of both local and remote databases.
It's fairly clear that you are going to implement the bulk of your filing system in user-space, but you will need a (small) kernel resident portion. Unless there's a really good reason to do otherwise, FUSE is what you're looking for - it will provide the kernel-resident part for you. All you'll need to write is glue to turn file operations into SQL requests.
The project is an open source IDE like system to edit code, alter images,, run code, etc. for students in school.
It contains an upload of Archive/Folders which characteristics are:
Upload Archive, contains up to 30 files (txt,js,xul,xml,php, gif/png/jpg), av.size in total 500kb
Users can edit and fork
all the files are editable through Codemirror (like used on jsfiddle)
all the Gif/Png/Jpg are replaceable (10 - 40 per archive)
we expect daily at least 1'000 new uploads/forks with an average of 20 files, total-size min. 500 mb
Our enironment:
PHP
most likely MySQL database
Linux
To consider:
We don't require Searching through the folders/files on global scope
the User saved Data is as it is, never any changes from our side necessary
State-of-Development:
Ready besides the storage question and all their dependencies
Would you advise on SQL or a simple Filesystem?
Before starting the project we were 100% sure using MySQL, but with the added feature of Image modification and a growing database (atm 80/k files,2GB) we struggle. Also reading here let us hesitate too.
Advantages of MYSQL are surely the easy maintainment and simpler future restructure of the system.
Though it will get a huge database fast to search within.
By using a global php entryfile to read the Filesystem based on URL parameters, the searching can be ommitted and straight go displaying the fetched directory with its content.
We are not experienced in managing large data and rely on experience of people who faced such a situtation already.
Rather than just vote for Database or Filesystem please consider your own tips to make this environment more efficient to run (eg indexing, structure, tables, etc..) or elaborate your decision.
very thankful for any tips of you
btw, the project will be hosted at git
I believe that the best option to go is the file system taking into consideration your requirements. The pros of this choice are
Better performance of your database since the table will only hold a link to your file system where the file is stored.
It is easier to backup your database.
You have the possibility of storing the files in different locations by defining rules. This gives you a flexibility in managing your storage.
When we faced a similar problem for storing attachments in a service desk application, we have chosen to go with the filesystem. Till now everything is working as expected.
I hope that my answer is helpful to you.
The application's code and configuration files are maintained in a code repository. But sometimes, as a part of the project, I also have a some data (which in some cases can be >100MB, >1GB or so), which is stored in a database. Git does a nice job in handling the code and its changes, but how can the development team easily share the data?
It doesn't really fit in the code version control system, as it is mostly large binary files, and would make pulling updates a nightmare. But it does have to be synchronised with the repository, because some code revisions change the schema (ie migrations).
How do you handle such situations?
We have the data and schema stored in xml and use liquibase to handle the updates to both the schema and the data. The advantage here is that you can diff the files to see what's going on, it plays nicely with any VCS and you can automate it.
Due to the size of your database this would mean a sizable "version 0" file. But, using the migration strategy, after that the updates should be manageable as they would only be deltas. You might be able to convert your existing migrations one-to-one to liquibase as well which might be nicer than a big-bang approach.
You can also leverage #belisarius' strategy if your deltas are very large so each developer doesn't have to apply the delta individually.
It seems to me that your database has a lot of parallels with a binary library dependency: it's large (well, much larger than a reasonable code library!), binary, and has its own versions which must correspond to various versions of your codebase.
With this in mind, why not integrate a dependency manager (e.g. Apache Ivy) with your build process and let it manage your database? This seems like just the sort of task that a dependency manager was built for.
Regarding the sheer size of the data/download, I don't think there's any magic bullet (short of some serious document pre-loading infrastructure) unless you can serialize the data into a delta-able format (the XML/JSON/SQL you mentioned).
A second approach (maybe not so compatible with dependency management): If the specifics of your code allow it, you could keep a second file that is a manual diff that can take a base (version 0) database and bring it up to version X. Every developer will need to keep a clean version 0. A pull (of a version with a changed DB) will consist of: pull diff file, copy version 0 to working database, apply diff file. Note that applying the diff file might take a while for a sizable DB, so you may not be saving as much time over the straight download as it first seems.
We usually use the database sync or replication schema.
Each developer has 2 copies of the database, one for working and the other just for keeping the sync version.
When the code is synchronized, the script syncs the database too (the central DB against the "dead" developer's copy). After that each developer updates his own working copy. Sometimes a developer needs to keep some of his/her data, so these second updates are not always driven by the standard script.
It is as robust as the replication schema .... sometimes (depending on the DB) that doesn't represent good news.
DataGrove is a new product that gives you version control for databases. We allow you to store the entire database (schema and data), tag, restore and share the database at any point in time.
This sounds like what you are looking for.
We're currently working on features to allow git-like (push-pull) behaviors so developers can share their repositories across machines, so I can load the latest version of your database when I need it.
I'm not really asking whether I should use either a RDBMS or config files for 100% of my application configuration, but rather what kind of configuration is best addressed by each method.
For example, I've heard that "any kind of configuration that is not changeable by the end-user" should be in config files rather than the database. Is this accurate? How do you address configuration?
(I'm primarily concerned with many-user web applications here, but no particular platform.)
I find that during development it is of great benefit to have configuration stored in a file.
It is far easier to check out a file (web.config, app.config, or some custom file) and make changes that are instantly picked up when the code is run. There is a little more friction involved in working with configuration stored in a database. If your team uses a single development database you could easily impact other team members with your change, and if you have individual databases it takes more than a "get latest" to be up and running with the latest configuration. Also, the flexibility of XML makes it more natural to store configuration that is more than just "name-value" pairs in a file than in a relational DB.
The drawback is where you want to reuse the configuration across multiple apps or web site instances. In my own case, we have a single config file in a well-known location that can be referenced by any application.
At least, this is how we store "static" configuration that does not have to be updated by the system at runtime. User settings are probably more suited to storage in the DB.
The oneliner: As a general principle - the more likely the config data should change the better to put it into db.
The legal disclaimer:
You would need to have almost always a kind of "bootstrapping" configuration, which must be saved into a file, thus if you are using a db to store your configuration the size of the "bootrapping" conf would depend on the other great principle:
"Work smarter not harder !!!"
One thing to conside is how much config data there is, and perhaps how often it is likely to change. If the amount of data is small, then saving this in a database (if your not already using a db for anything else), would be overkill, equally maintaining a db for something that gets changed once every 6 months would probably be a waste of resources.
That said, if your already using a database for other parts of your site, then adding a table or two for configuration data is probabley not a big issue, and may fit in well with the way you are storing the rest of your data. If you already have a class for saving your data to a db, why write a new one to save to a config file.