I've been tasked with transferring our information out of an old antiquated system, back into Access.
I have personally not used Access before, and I'm seeing that in order to transfer the data, I have to compact the file, because the data set was created on an older version of Access.
After hours of trying to find a function to do this that actually works, it seems like I can't find anything that works. Has anyone had success in this?
I'm wondering if it's easier to have two data sets, one in excel that is searchable for the old data set, and create a new database for all new items in Access, and anything that needs to be changed from the old database, could be transferred on a "as needed" basis.
We track Serial Numbers, Warranty & Purchase dates, Purchase Order Numbers, along with customer names. So it's not a large amount of information, however it needs to be clear and efficient.
Any wise words greatly appreciated.
Related
I am trying to realize a project where people can login into a site where they find a personal calendar. In this calendar people shall be able to leave timestamps. Since a year has around 365 days, (and per day more than 1 timestamp is possible) there will be a lot of timestamps to save.
I need a way to save those timestamps in a sort of database. I am new to this and I want to know if using a JSON File for storing those timestamps or using a MySQL database is the better way of doing this.
Background-Story:
I work on a project where a microcontroller does certain things at those given timestamps from the User. The user leaves timestamps in a calendar on an iOS-App. So it also has to be compatible with Swift/iOS.
Any ideas?
Databases have a few ways to store timestamps. For example the data-type TIMESTAMP or DATETIME are all ways to store timestamps
If you do it in a database, you have the ability to sync it across devices.
To do it in JSON, I'll refer you to this question on StackOverflow:
The "right" JSON date format
EDIT: After reviewing the comments, you most likely want a database. I have an example here where you have a table for users and a table for events that can be joined to get each event for each user, even though each user has all their events in the same table.
I created this because I used to not know what Databases were good for, so I came here and someone put me in the right direction. Databases are VERY powerful and fast. To maintain everyone's JSON file of events would be a nightmare. I 100% recommend a database for your situation
Play around with this sample DB I created: http://sqlfiddle.com/#!9/523e2d/5
If you have only a few users with a few timestamps and not much else going on, then you could indeed store each user’s data in a text file. That file could store JSON if you want, or XML, or more simply tab-delimited or comma-delimited text. If you just need the next timestamp, without keeping history, this might well be the best approach with one file per user. Beware of risks such as your own code inadvertently overwriting a file, or crashing during a write so the file is ruined and data lost.
A step-up from text-in-a-file is SQLite, to help you manage the data so that you need not do the chore of parsing. While SQLite is a valuable product when used appropriately, it is not meant to compete with serious database products.
But for more heavy-duty needs, such as much more data or multiple apps accessing the data, use a database. That is what they are for – storing large amounts of structured data in a way that can handle concurrent access by more than one user and more than one app.
While MySQL is more famous, for serious enterprise-quality database where preserving your data is critical, I recommend Postgres. Postgres has the best date-time handling of most any database, both in data types and in functions. Postgres also has the best documentation of any SQL database.
Another pair of options, both built in pure Java and both free-of-cost/open-source, are H2 and Derby. Or commercial products like Microsoft SQL Server or Oracle.
You have not really given enough details to make a specific recommendation. Furthermore, this site is not meant for software recommendations. For that, see the sister site: Software Recommendations Stack Exchange.
If you choose a database product, be sure to learn about date-time handling. Database products vary widely in this respect, and the SQL standard barely touches upon the subject. Quick tips: Store values in UTC, and indeed your database such as Postgres is likely to adjust incoming values. Apply a time zone only for presentation to users. Avoid the data-types that ignore the issue of time zone. So for standard SQL, this means using the TIMESTAMP WITH TIME ZONE data type rather than TIMESTAMP WITHOUT TIME ZONE. For Java work, use only the java.time classes rather than the troublesome old date-time classes, and for your purposes use the Instant and ZonedDateTime classes but not the LocalDateTime class which lacks any offset or time zone.
I am new to database development so any advice is appreciated.
I am tasked with creating a SQL database. This database will store data that is currently on an Excel file. The data in about 50 rows changes every single day (different rows each day).
The person updating the data is a non-programmer and the updating of info needs to be simple and fast.
I'm thinking my client could just update the Excel file, and then this Excel file will upload to the database. Is that feasible? Is there a better way? My client spends enough time just updating the Excel file, so anything that will take a significant amount of extra time for inputing data is not feasible.
Any feedback or ideas is appreciated!
Context: I haven't made any decisions about which SQL DBMS I will use (or maybe I'll use noSQL or Access). I'm still in the how-should-I-approach-this stage of development.
If your data all fits in an excel file, there's nothing wrong with that approach. You need to spend your time thinking about how you want to get the data from excel in to the DB, as you have a ton of options as far as programming languages / tools to do that.
I'm personally a huge fan of Node.js (there are npm modules already existing for reading excel files, writing to mysql, etc, so you would end up writing almost nothing yourself) but you could do it using just about anything.
Do yourself a favor and use a simple database (like MySQL) and don't mess with NoSQL for this. The amount of data you have is tiny (if it's coming from an excel file) and you really don't need to start worrying about the complexities of NoSQL until you have a a TON of data or data that is changing extremely rapidly.
If the business process is currently updating Excel, then that is probably the best approach. Modify the Excel file. A little bit of VBA in the Excel can copy it and store it somewhere for the database. Schedule a job in the evening, load the data in, and do whatever processing you like. This is not an unusual architecture.
Given your choice of tools, I would advise you not to use either Access or a NoSQL database, unless you have a particular reason for choosing them. A free database such as Postgres or SQL Server Express should do a good job meeting your needs.
I have a website on a shared host, where I expect a lot of visitors. I don't need a database for reading (everything presented on the pages is hardcoded in PHP) but I would like to store data that my users enter, so for writing only. In fact, I only store this to do a statistical analysis on it afterwards (on my local computer, after downloading it).
So my two questions:
Is MySQL a viable option for this? It is meant to run on shared hosting, with PHP/MySQL available, so I cannot really use much other fancy packages, but if e.g. writing to a file would be better for this purpose, that's possible too. As far as I understood, adding a line to a file is a relatively complex operation for huge files. On the other hand, 100+ users simultaneously connecting to a MySQL database is probably also a huge load, even if it's just for doing 1 cheap INSERT query.
If MySQL is a good option, how should the table best be configured? Currently I have one InnoDB table, with a primary key id that auto-increments (next to of course the columns storing the data). This is general-purpose configuration, so maybe there are more optimized ways given that I only need to write to the table, and not read from it?
Edit: I mainly fear that the website will go viral as soon as it's released, so I expect the users to visit in a very short timeframe. And of course I would not like to lose the data that they enter due to an overloaded database.
MySQL is a perfectly reasonable choice for this. Probably much better than a flat file, since you say you want to aggregate and analyze this data later. Doing so with a flat file might take a long time, especially if the file is large. Additionally, RDBMS are for aggregation and dataset manipulation. Ideal for creating report data.
Put whatever data columns you want in your table, and some kind of identifier to track a user, in addition to your existing row key. IP address is a logical choice for user tracking, or a magic cookie value could potentially work. It's only a single table, you don't need to think too hard about it. You may want to add nonclustered indexes on columns you'll frequently filter on for reports, e.g. IP address, access date, etc.
I mainly fear that the website will go viral as soon as it's released, so I expect the users to visit in a very short timeframe. And of course I would not like to lose the data that they enter due to an overloaded database.
RDBMS such as MySQL are explicitly designed to handle heavy loads, assuming appropriate hardware backing. Don't sweat it.
Our company has a database of 17,000 entries. We have used MS Access for over 10 years for our various mailings. Is there something new and better out there? I'm not a techie, so keep in mind when answering. Our problems with Access are:
-no record of what was deleted,
-will not turn up a name in a search if cap's or punctuation
is not entered exactly,
-is complicated for us to understand the de-duping process.
- We'd like a more nimble program that we can access from more than one dedicated computer.
The only applications I know of that are comparable to Access are FileMaker Pro, and the database component of the Open Office suite. FM Pro is a full-fledged product and gets good marks for ease of use from non-technical users, while Base is much less robust and is not nearly as easy for creating an application.
All of the answers recommending different databases really completely miss the point here -- the original question is about a data store and application builder, not just the data store.
To the specific problems:
PROBLEM 1: no record of what was deleted
This is a design error, not a flaw in Access. There is no database that really keeps a record of what's deleted unless someone programs logging of deleted data.
But backing up a bit, if you are asking this question it suggest that you've got people deleting things that shouldn't be deleted. There are two solutions:
regular backups. That would mean you could restore data from the last backup and likely recover most of the lost data. You need regular backups with any database, so this is not really something that is specific to Access.
design your database so records are never deleted, just marked deleted and then hidden in data entry forms and reports, etc. This is much more complicated, but is very often the preferred solution, as it preserves all the data.
Problem #2: will not turn up a name in a search if cap's or punctuation is not entered exactly
There are two parts to this, one of which is understandable, and the other of which makes no sense.
punctuation -- databases are stupid. They can't tell that Mr. and Mister are the same thing, for instance. The solution to this is that for all data that needs to be entered in a regularized fashion, you use all possible methods to insure that the user can only enter valid choices. The most common control for this is a dropdown list (i.e., "combo box"), which limits the choices the user has to the ones offered in the list. It insures that all the data in the field conforms to a finite set of choices. There are other ways of maintaining data regularity and one of those involves normalization. That process avoids the issue of repeatedly storing, say, a company name in multiple records -- instead you'd store the companies in a different table and just link your records to a single company record (usually done with a combo box, too). There are other controls that can be used to help insure regularity of data entry, but that's the easiest.
capitalization -- this one makes no sense to me, as Access/Jet/ACE is completely case-insensitive. You'll have to explain more if you're looking for a solution to whatever problem you're encountering, as I can't conceive of a situation where you'd actually not find data because of differences in capitalization.
Problem #3: is complicated for us to understand the de-duping process
De-duping is a complicated process, because it's almost impossible for the computer to figure out which record among the candidates is the best one to keep. So, you want to make sure your database is designed so that it is impossible to accidentally introduce duplicate records. Indexing can help with this in certain kinds of situations, but when mailing lists are involved, you're dealing with people data which is almost impossible to model in a way where you have a unique natural key that will allow you to eliminate duplicates (this, too, is a very complicated topic).
So, you basically have to have a data entry process that checks the new record against the existing data and informs the user if there's a duplicate (or near match). I do this all the time in my apps where the users enter people -- I use an unbound form where they type in the information that is the bare minimum to create a new record (usually some combination of lastname, firstname, company and email), and then I present a list of possible matches. I do strict and loose matching and rank by closeness of the match, with the closer matches at the top of the list.
Then the user has to decide if there's a match, and is offered the opportunity to create the duplicate anyway (it's possible to have two people with the same name at the same company, of course), or instead to abandon adding the new record and instead go to one the existing records that was presented as a possible duplicate.
This leaves it up to the user to read what's onscreen and make the decision about what is and isn't a duplicate. But it maximizes the possibility of the user knowing about the dupes and never accidentally creating a duplicate record.
Problem #4: We'd like a more nimble program that we can access from more than one dedicated computer.
This one confuses me. Access is multi-user out of the box (and has been from the very beginning, nearly 20 years ago). There is no limitation whatsoever to a single computer. There are things you have to do to make it work, such as splitting your database into to parts, one part with just the data tables, and the other part with your forms and reports and such (and links to the data tables in the other file). Then you keep the back end data file on one of the computers that acts as a server, and give a copy of the front end (the reports, forms, etc.) to each user. This works very well, actually, and can easily support a couple of dozen users (or more, depending on what they are doing and how well your database is designed).
Basically, after all of this, I would tend to second #mwolfe02's answer, and agree with him that what you need is not a new database, but a database consultant who can design for you an application that will help you manage your mailing lists (and other needs) without you needing to get too deep into the weeds learning Access (or FileMaker or whatever). While it might seem more expensive up front, the end result should be a big productivity boost for all your users, as well as an application that will produce better output (because the data is cleaner and maintained better because of the improved data entry systems).
So, basically, you either need to spend money upfront on somebody with technical expertise who would design something that allows you to do better work (and more efficiently), or you need to invest time in upping your own technical skills. None of the alternatives to Access are going to resolve any of the issues you've raised without significant investment in interface design to further the goals you have (cleaner data, easier to find information, etc.).
At the risk of sounding snide, what you are really looking for is a consultant.
In the hands of a capable programmer, all of your issues with Access are easily handled. The problems you are having are not the result of using the wrong tool, but using that tool less than optimally.
Actually, if you are not a techie then Access is already the best tool for you. You will not find a more non-techie friendly way to build a data application from bottom to top.
That said, I'd say you have three options at this point:
Hire a competent database consultant to improve your application
Find commercial off-the-shelf (COTS) software that does what you need (I'm sure there are plenty of products to handle mailings; you'll need to research)
Learn about database normalization and building proper MS Access applications
If you can find a good program that does what you want then #2 above will maximize your Return on Investment (ROI). One caveat is that you'll need to convert all of your existing data, which may not be easy or even possible. Make sure you investigate that before you buy anything.
While it may be the most expensive option up-front, hiring a competent database consultant is probably your best option if you need a truly custom solution.
SQL Server sounds like a viable alternative to your scenario. If cost is a concern, you can always use SQL Server Express, which is free. Full blown SQL Server provides a lot more functionality that might not be needed right away. Express is a lot simpler as the number of features provided with it are much smaller. With either version though you will have centralized store for your data and the ability to allow all transactions to be recorded in the transaction log. Also, both have the ability to import data from an Access Database.
The newest version of SQL Server is 2008 R2
You probably want to take a look at modern databases. If you're into Microsoft-based products, start with SQL Server Express
EDIT: However, since I understand that you're not a programmer yourself, you'd probably be better off having someone experienced look into your technical problem more deeply, like the other answer suggests.
It sounds like you may want to consider a front-end for your existing Access data store. Microsoft has yet to replace Access per se, but they do have a new tool that is a lot lower on the programming totem pole than some other options. Check out Visual Studio Lightswitch - http://www.microsoft.com/visualstudio/en-us/lightswitch.
It's fairly new (still in beta) but showing potential. With it, just as with any Visual Studio project, you can connect to an MS Access datasource and design a front-end to interact with it. The plus-side here is programming requirements are much lower than with straight-up Visual Studio (read: Wizards).
Given that replacing your Access DB will require some font-end programming, you may look into VistaDB. It should allow your front end to be created in .NET with an XCopy database on the backend without requiring a server. One plus is that it retains SQL Server syntax, so if you do decide to move to SQL Server you'll be one step ahead.
(Since you're not a techie and may not understand my previous statement, you might pass my answer on to the consultant/programmer/database guy who is going to do the work for you.)
http://www.vistadb.net/
I need a concept how I can archive all data in D-Pool which is older than one year.
At the moment we have more than 3 million records in the D-Pool.
Because of this huge data foundation searches and filters over the database takes quite too long, because most searches are done over the whole D-Pool data, but in most cases I am only interested in current data.
So I want to archive regularly all data from D-pool which is not needed for current work and evaluations.
But for some functions it should be possible furthermore to do a search over the whole D-pool, the current and the old data.
Could you tell me some ideas for this problem?
This describes the typical data warehousing solution. Most owners of large datasets that change daily have a transactional database and a historical or reporting database. The historical or reporting database allows users to mine for insight against everything - except the data added since the last extract. That's usually sufficient.