I'm trying to develop a new reporting module for a resource management tool (PHP+Mysql).
I am trying to extract data in the following format from mysql:
I have a table that consists of date and location of multiple people(i.e Office, Home or Client).
Sample Data as in DB.
here date_plotted means the date at which the user is engaged and plotting_date represents when this particular entry was made in the system(the date). So User was plotted to be in office on 30th Oct and the same entry was made on 30th Oct.
Data as in resource table
The resource table represents the user table.
Any suggestions on how to do the same in mysql?
These are the primary tables which needs to be used.
The above table id done in excel for now to represent the outcome.
I'm new to SQL so haven't tried anything yet.
There is a tool for Windows that might simplify this operation. It's made by MySQL and called MySQL for Excel. In theory it should allow you to structure and make changes to MySQL databases as well as perform queries that result in spreadsheets.
Without knowing more about your data, for example being supplied an actual csv file to work with, and the parameters of the actual pull, whether it's fix dates always or if this is a dynamic pull based on a range this question could result in 100 different implementations that visually return similar results, but have massively different requirements overhead-wise in implementation.
Related
On our Wordpress site, we use a plugin called s2member and it stores the levels (roles) of our clients as well as the times they were assigned a specific level in our database. I would like to create a table that shows when a user was assigned a specific level. I'm having a challenge getting the data I need because of the way the data is stored in the field. It stores all of the levels along with the associated dates and times when a user's level was changed in one field. In addition, it stores all of the times as Unix timestamps. Here's an example of a typical field associated with a client:
a:20:{s:15:"1562695223.0001";s:6:"level0";s:15:"1562695223.0002";s:6:"level1";s:15:"1562695223.0003";s:6:"level2";s:15:"1562695223.0004";s:6:"level3";s:15:"1577906312.0001";s:11:"ccap_prepay";s:15:"1596575898.0001";s:12:"-ccap_prepay";s:15:"1596575898.0002";s:13:"ccap_graduate";s:15:"1596575898.0003";s:11:"ccap_prepay";s:15:"1596575898.0004";s:7:"-level3";s:15:"1597196952.0001";s:14:"-ccap_graduate";s:15:"1597196952.0002";s:12:"-ccap_prepay";s:15:"1597196952.0003";s:13:"ccap_graduate";s:15:"1597196952.0004";s:11:"ccap_prepay";s:15:"1598382433.0001";s:14:"-ccap_graduate";s:15:"1598382433.0002";s:12:"-ccap_prepay";s:15:"1598382433.0003";s:11:"ccap_prepay";s:15:"1598382433.0004";s:6:"level3";s:15:"1605290551.0001";s:12:"-ccap_prepay";s:15:"1605290551.0002";s:11:"ccap_prepay";s:15:"1605290551.0003";s:13:"ccap_graduate";}
There are four columns in this table: umeta_id; user_id; meta_key; meta_value. The data above is stored in the meta_value column.
You'll notice that it also has multiple ccap_* entries. CCAP stands for custom capapability and I would like to be able to chart those assignments and associated times as well.
Do you have any idea how I can accomplish this?
Thank you for any help you can give.
I talked to an engineer about this and he told me that I would need to learn Python and I believe he said I would need to learn how to also use Pandas and Numpy to extract the data I need but he wasn't exactly sure. I started taking a data analyst course on Coursera but I still haven't learned what I need to learn and it's already been several months. It would be great if someone could provide a solution that I could implement more quickly and use on an ongoing basis.
If there's a way to accomplish my goal by exporting this table to a CSV file and using Microsoft Excel or Google Sheets, I'm open to that too.
Here's an image of the table (if it helps):
Database table
Here's an example of my desired output:
Desired output
In my desired output, I used Excel and created a column that converts the Unix timestamp to a short date and another column where I used a nested IF statement to convert the CCAP or level to its meaning that we understand internally.
In a data warehouse project how do I verify that my fact table loaded in a data warehouse DB through SSIS ETL load is correct with my staging table so that later I don't have incorrect reporting?
Good question, people creates different systems for this. So you understand this is one of most complex check/reconciliation process that developers built. I tried to give you three ways to do this. I would recommend first one because its easier and most efficient.
You can -
Post Load reports: create reports which will reconcile data after load. Write SQL to compare source data and target data - compare count, compare amount, compare null values, compare daily data etc. If the comparison generates flag/alert - this means some issue in load.
Check as you go : You can create some reusable function or mapping which will compare incoming source data and target data - compare count, compare amount, compare null values, compare daily data etc. and store in a table. A script will keep on checking those values and if there is any issue, script will notify support team.
Pre process check : Before starting any ETL, you can check source data - like count, null values, daily count etc. to verify how is the data, if there is any file missing etc.
I am looking for a way to store auto-generated reports. There are about 10-15 columns and 100-3000 rows depending on the report but each report is consistent in column count.
I am looking for a way to organise and store these reports into a large group without creating an entire new database and 1000s of tables to store each indervidual report.
The reports need to be queryable so they can be subdivided by team/area/person etc as each report can be a combination of 3-4 different sub-reports depending on how you split/sort the data.
I am using Python to collect and sort the data from the database so using MariaDB/MySQL would be preferred but im happy to use something else if there is a pre-exising connection libary for it.
To sum up i need something similar to a excel spreadsheet with each table being a sheet and sheet name being the date it was generated so i can select by the date generated.
Think through the goals.
Is this a legal issue -- you need to produce an unalterable report as something "official". A la a non-editable .pdf?
(at the opposite extreme) Be able to generate (or regenerate) any report for any timeframe.
Is performance an issue? (Either perceived or real)
I like to build and maintain Summary Table(s) for any "Data Warehouse" application. And build "reports" that take as a parameter a date range and a small number of other things. And have the report generation so fast that it does not matter if multiple people are pulling reports at random times.
15 columns and 3000 rows is usually excessive. If pulling a report is trivial enough, it can be less 'massive'; just get the parts you want, without such bulk.
http://mysql.rjweb.org/doc.php/summarytables
I need to create a database which allow for easy manual entry of data as well as linking to SQL databases I have access to. This is something I've been researching and trying to design by myself for a while now. Everytime I think I have a solution I begin to have doubts that this is the right direction. I would really REALLY appreciate your help/input.
My current workflow and nightmare.
I work in IT and I'm responsible for reporting for my own group but it has been and will continue to expand across many groups across the whole IT organization. Each group has their own reporting tool with some that can export data cleanly/consistently, others that have to input data manually and a few which I can access the SQL database directly but haven't yet.
To create a monthly report I have a master workbook with graphs that reference data in multiple data collection workbooks. In these data collection workbooks I have the raw data which either myself or the team/service owners inputs each month. From that raw data I may create a pivot table to get the data I want and then in each one of the collection workbooks I have a 'reference" tab. In this reference tab I have a formula that is a rolling 13 months in the first column which say goes from Jan 2013 to Jan 2014. The months are calculated by referencing a cell that has the current reporting month in it. So for this example its Jan 2014. When you change that cell to the next month say Feb 2014 it will roll that first column so that those same cells now go from Feb 2013 to Jan 2014. the columns to the right of this column reference either the data table directly or the pivot tables using vlookups() or getpivotdata() formulas that reference the month cell to the left and grabs data accordingly. Here is an example of a collection workbook.
As you can see this can be a very manual and tedious workflow with many points where this could be improved. One major issue is the fact that each team/group doesn't pull data from the same tool and doesn't serve it up in the right format. I sometimes have to write a vba script to take the raw data and add helper data like a column with the correctly formatted 02/01/2014 format so it works in pivot tables correctly instead of multiple entries with different days (ie. 02/05/2014 02/23/2014).
The end goal + where I want to be
I want to create a database where I can link SQL databases directly to it as well as making it easy for a team to manually input data by reading in a csv or xls file of data. The crux again is that I'm only one person and I want to offload as much as the data entry to each one of those teams/groups without making it a difficult task.
Using the master sheet to create a monthly report has some great results but I would like this to be a web accessible and dynamic dashboard. I'm a pretty good front end dev with knowledge of HTML, CSS+LESS and some JS+SQL knowledge. I will have support of a senior web developer if I can proceed with this project. I just need help in working out the data collection/storage/querying portion of this.
If you got this far thank you so much for reading about my problem. If I wasn't clear about something or you have more questions for me please feel free to ask.
THANKS!
You may want to take a look at SQL Server's linked servers, according to the documentation, you can link any database who have an OLE DB driver so you can run queries mixing SQL Server and other databases, so it may be worth your while to check this.
I have an LDAP CSV file that is imported nightly and dumped into my MYSQL database. It has about 70000 employee records.
Included in that is empl#, email, group, supervisor, etc.
I have reports that are being generated from various web sites. We are dumping these reports in the database once a month. These reports usually have empl#, email, hits, logins, whatever...
My goal is to combine the report data and add in things like group, supervisor, etc based on empl#... Speed is a big concern because of the size of the database and number of users.
At first I thought of making a simple left join (given that report data is left - and that all people in the report may not be an employee). However the problem with that is that it does not take a snapshot in time. If report data from 6 months ago is viewed I don't want it mixed with current employee data - I want it to stay a snapshot in time.
What is the best way to handle this?
You will need a date column of some kind in both sets of data on which to join. Once you have that, you can simply put a condition that establishes the snapshot in the WHERE that limits the selection.