How to split one excel table to several MySQL tables? - mysql

I am working for an organization that collects data from primary care physicians, and so we have very little input on how the data arrives here.
The physicians send us an excel sheet with ~40 columns, only about 5 of which they use for any given appointment, depending on what that particular appointment is for.
I want to normalize our database, but to avoid that creating enormous friction in uploading our data, I would have to find a way to upload from the excel file into the variously split files that are appropriate for the kind of appointment that was conducted (effectively, specified by the null columns), without requiring human oversight to sort every appointment before upload.
Is there any convenient way to do this?

Related

How to Filter Data in a Single MySQL Database Field that has Multiple Entries

On our Wordpress site, we use a plugin called s2member and it stores the levels (roles) of our clients as well as the times they were assigned a specific level in our database. I would like to create a table that shows when a user was assigned a specific level. I'm having a challenge getting the data I need because of the way the data is stored in the field. It stores all of the levels along with the associated dates and times when a user's level was changed in one field. In addition, it stores all of the times as Unix timestamps. Here's an example of a typical field associated with a client:
a:20:{s:15:"1562695223.0001";s:6:"level0";s:15:"1562695223.0002";s:6:"level1";s:15:"1562695223.0003";s:6:"level2";s:15:"1562695223.0004";s:6:"level3";s:15:"1577906312.0001";s:11:"ccap_prepay";s:15:"1596575898.0001";s:12:"-ccap_prepay";s:15:"1596575898.0002";s:13:"ccap_graduate";s:15:"1596575898.0003";s:11:"ccap_prepay";s:15:"1596575898.0004";s:7:"-level3";s:15:"1597196952.0001";s:14:"-ccap_graduate";s:15:"1597196952.0002";s:12:"-ccap_prepay";s:15:"1597196952.0003";s:13:"ccap_graduate";s:15:"1597196952.0004";s:11:"ccap_prepay";s:15:"1598382433.0001";s:14:"-ccap_graduate";s:15:"1598382433.0002";s:12:"-ccap_prepay";s:15:"1598382433.0003";s:11:"ccap_prepay";s:15:"1598382433.0004";s:6:"level3";s:15:"1605290551.0001";s:12:"-ccap_prepay";s:15:"1605290551.0002";s:11:"ccap_prepay";s:15:"1605290551.0003";s:13:"ccap_graduate";}
There are four columns in this table: umeta_id; user_id; meta_key; meta_value. The data above is stored in the meta_value column.
You'll notice that it also has multiple ccap_* entries. CCAP stands for custom capapability and I would like to be able to chart those assignments and associated times as well.
Do you have any idea how I can accomplish this?
Thank you for any help you can give.
I talked to an engineer about this and he told me that I would need to learn Python and I believe he said I would need to learn how to also use Pandas and Numpy to extract the data I need but he wasn't exactly sure. I started taking a data analyst course on Coursera but I still haven't learned what I need to learn and it's already been several months. It would be great if someone could provide a solution that I could implement more quickly and use on an ongoing basis.
If there's a way to accomplish my goal by exporting this table to a CSV file and using Microsoft Excel or Google Sheets, I'm open to that too.
Here's an image of the table (if it helps):
Database table
Here's an example of my desired output:
Desired output
In my desired output, I used Excel and created a column that converts the Unix timestamp to a short date and another column where I used a nested IF statement to convert the CCAP or level to its meaning that we understand internally.

MySQL Storing Reports

I am looking for a way to store auto-generated reports. There are about 10-15 columns and 100-3000 rows depending on the report but each report is consistent in column count.
I am looking for a way to organise and store these reports into a large group without creating an entire new database and 1000s of tables to store each indervidual report.
The reports need to be queryable so they can be subdivided by team/area/person etc as each report can be a combination of 3-4 different sub-reports depending on how you split/sort the data.
I am using Python to collect and sort the data from the database so using MariaDB/MySQL would be preferred but im happy to use something else if there is a pre-exising connection libary for it.
To sum up i need something similar to a excel spreadsheet with each table being a sheet and sheet name being the date it was generated so i can select by the date generated.
Think through the goals.
Is this a legal issue -- you need to produce an unalterable report as something "official". A la a non-editable .pdf?
(at the opposite extreme) Be able to generate (or regenerate) any report for any timeframe.
Is performance an issue? (Either perceived or real)
I like to build and maintain Summary Table(s) for any "Data Warehouse" application. And build "reports" that take as a parameter a date range and a small number of other things. And have the report generation so fast that it does not matter if multiple people are pulling reports at random times.
15 columns and 3000 rows is usually excessive. If pulling a report is trivial enough, it can be less 'massive'; just get the parts you want, without such bulk.
http://mysql.rjweb.org/doc.php/summarytables

MySQL: Dates as column headers

I'm trying to develop a new reporting module for a resource management tool (PHP+Mysql).
I am trying to extract data in the following format from mysql:
I have a table that consists of date and location of multiple people(i.e Office, Home or Client).
Sample Data as in DB.
here date_plotted means the date at which the user is engaged and plotting_date represents when this particular entry was made in the system(the date). So User was plotted to be in office on 30th Oct and the same entry was made on 30th Oct.
Data as in resource table
The resource table represents the user table.
Any suggestions on how to do the same in mysql?
These are the primary tables which needs to be used.
The above table id done in excel for now to represent the outcome.
I'm new to SQL so haven't tried anything yet.
There is a tool for Windows that might simplify this operation. It's made by MySQL and called MySQL for Excel. In theory it should allow you to structure and make changes to MySQL databases as well as perform queries that result in spreadsheets.
Without knowing more about your data, for example being supplied an actual csv file to work with, and the parameters of the actual pull, whether it's fix dates always or if this is a dynamic pull based on a range this question could result in 100 different implementations that visually return similar results, but have massively different requirements overhead-wise in implementation.

Access: Inventory Management with 2 seperate criteria

im a bit of a newbie in access and i hope im not asking a stupid question. I have recently had to move an inventory system from excel to access. Each product is recipted in tbl.rct and has an order number a lot number quantity and expiry.
Each individual lot number needs to be verified before it can be recipted this information is on tbl.lot.
While making a form to receipt products i noticed that i couldnt add any products without their lot number already on lot.tbl - is there a way to get around this?
http://imgur.com/kCc7G39
Attached relationships
I think you mix between Excel and the Access. These Table imported directly from the excel without any requirement change to meet Access Goals. The Database use to reduce the repeat routine work. The Tables that most be (Products, Order, Receipts, Lot must be Stock and collect data of (Qty, lot#, expiry, damage). Now we make sequence to how insert to Database. Open New Receipt to include in the stock the Product(link ID) and the detail. This is now in the warehouse. For selling you will make invoice when select product will show you the Lot available and its expiry and of course you select filter to filter on FIFP LIFO.
You can send me the excel file to convert to database if yes please provide me more information because the flow not clear well

Merging table data in mysql?

I have an LDAP CSV file that is imported nightly and dumped into my MYSQL database. It has about 70000 employee records.
Included in that is empl#, email, group, supervisor, etc.
I have reports that are being generated from various web sites. We are dumping these reports in the database once a month. These reports usually have empl#, email, hits, logins, whatever...
My goal is to combine the report data and add in things like group, supervisor, etc based on empl#... Speed is a big concern because of the size of the database and number of users.
At first I thought of making a simple left join (given that report data is left - and that all people in the report may not be an employee). However the problem with that is that it does not take a snapshot in time. If report data from 6 months ago is viewed I don't want it mixed with current employee data - I want it to stay a snapshot in time.
What is the best way to handle this?
You will need a date column of some kind in both sets of data on which to join. Once you have that, you can simply put a condition that establishes the snapshot in the WHERE that limits the selection.