I need to create a database which allow for easy manual entry of data as well as linking to SQL databases I have access to. This is something I've been researching and trying to design by myself for a while now. Everytime I think I have a solution I begin to have doubts that this is the right direction. I would really REALLY appreciate your help/input.
My current workflow and nightmare.
I work in IT and I'm responsible for reporting for my own group but it has been and will continue to expand across many groups across the whole IT organization. Each group has their own reporting tool with some that can export data cleanly/consistently, others that have to input data manually and a few which I can access the SQL database directly but haven't yet.
To create a monthly report I have a master workbook with graphs that reference data in multiple data collection workbooks. In these data collection workbooks I have the raw data which either myself or the team/service owners inputs each month. From that raw data I may create a pivot table to get the data I want and then in each one of the collection workbooks I have a 'reference" tab. In this reference tab I have a formula that is a rolling 13 months in the first column which say goes from Jan 2013 to Jan 2014. The months are calculated by referencing a cell that has the current reporting month in it. So for this example its Jan 2014. When you change that cell to the next month say Feb 2014 it will roll that first column so that those same cells now go from Feb 2013 to Jan 2014. the columns to the right of this column reference either the data table directly or the pivot tables using vlookups() or getpivotdata() formulas that reference the month cell to the left and grabs data accordingly. Here is an example of a collection workbook.
As you can see this can be a very manual and tedious workflow with many points where this could be improved. One major issue is the fact that each team/group doesn't pull data from the same tool and doesn't serve it up in the right format. I sometimes have to write a vba script to take the raw data and add helper data like a column with the correctly formatted 02/01/2014 format so it works in pivot tables correctly instead of multiple entries with different days (ie. 02/05/2014 02/23/2014).
The end goal + where I want to be
I want to create a database where I can link SQL databases directly to it as well as making it easy for a team to manually input data by reading in a csv or xls file of data. The crux again is that I'm only one person and I want to offload as much as the data entry to each one of those teams/groups without making it a difficult task.
Using the master sheet to create a monthly report has some great results but I would like this to be a web accessible and dynamic dashboard. I'm a pretty good front end dev with knowledge of HTML, CSS+LESS and some JS+SQL knowledge. I will have support of a senior web developer if I can proceed with this project. I just need help in working out the data collection/storage/querying portion of this.
If you got this far thank you so much for reading about my problem. If I wasn't clear about something or you have more questions for me please feel free to ask.
THANKS!
You may want to take a look at SQL Server's linked servers, according to the documentation, you can link any database who have an OLE DB driver so you can run queries mixing SQL Server and other databases, so it may be worth your while to check this.
Related
On our Wordpress site, we use a plugin called s2member and it stores the levels (roles) of our clients as well as the times they were assigned a specific level in our database. I would like to create a table that shows when a user was assigned a specific level. I'm having a challenge getting the data I need because of the way the data is stored in the field. It stores all of the levels along with the associated dates and times when a user's level was changed in one field. In addition, it stores all of the times as Unix timestamps. Here's an example of a typical field associated with a client:
a:20:{s:15:"1562695223.0001";s:6:"level0";s:15:"1562695223.0002";s:6:"level1";s:15:"1562695223.0003";s:6:"level2";s:15:"1562695223.0004";s:6:"level3";s:15:"1577906312.0001";s:11:"ccap_prepay";s:15:"1596575898.0001";s:12:"-ccap_prepay";s:15:"1596575898.0002";s:13:"ccap_graduate";s:15:"1596575898.0003";s:11:"ccap_prepay";s:15:"1596575898.0004";s:7:"-level3";s:15:"1597196952.0001";s:14:"-ccap_graduate";s:15:"1597196952.0002";s:12:"-ccap_prepay";s:15:"1597196952.0003";s:13:"ccap_graduate";s:15:"1597196952.0004";s:11:"ccap_prepay";s:15:"1598382433.0001";s:14:"-ccap_graduate";s:15:"1598382433.0002";s:12:"-ccap_prepay";s:15:"1598382433.0003";s:11:"ccap_prepay";s:15:"1598382433.0004";s:6:"level3";s:15:"1605290551.0001";s:12:"-ccap_prepay";s:15:"1605290551.0002";s:11:"ccap_prepay";s:15:"1605290551.0003";s:13:"ccap_graduate";}
There are four columns in this table: umeta_id; user_id; meta_key; meta_value. The data above is stored in the meta_value column.
You'll notice that it also has multiple ccap_* entries. CCAP stands for custom capapability and I would like to be able to chart those assignments and associated times as well.
Do you have any idea how I can accomplish this?
Thank you for any help you can give.
I talked to an engineer about this and he told me that I would need to learn Python and I believe he said I would need to learn how to also use Pandas and Numpy to extract the data I need but he wasn't exactly sure. I started taking a data analyst course on Coursera but I still haven't learned what I need to learn and it's already been several months. It would be great if someone could provide a solution that I could implement more quickly and use on an ongoing basis.
If there's a way to accomplish my goal by exporting this table to a CSV file and using Microsoft Excel or Google Sheets, I'm open to that too.
Here's an image of the table (if it helps):
Database table
Here's an example of my desired output:
Desired output
In my desired output, I used Excel and created a column that converts the Unix timestamp to a short date and another column where I used a nested IF statement to convert the CCAP or level to its meaning that we understand internally.
I am looking for solution to taking a template excel document and embedding it onto my website. After this users will use the template filling out certain cells. Is it possible for me to scrape a embedded excel sheet and use POST methods to send the data to my SQL Server database.
I want to embed the template because I need to change it from time to time, which means I love to have it easily changed within my code using the embed and iframe.
I have looked online and haven't found any solid documentation that this is possible.
I suggest that you start by thinking about the differences between data in an Excel spreadsheet and data in a normalised SQL Server database.
For example, suppose you have a spreadsheet that you use to record downloads of "digital items" such as pdf files, word documents, .zip files, and so on.
Each column represents a date and each row represents a digital item.
So lest say Row 4 is for "The Cat Story.pdf" and Column 3 is for "15 February 2019. and Column 4 is for 16 February 2019 and so on. The value in Cell Row 4-Column 3 is "206", and the value in Cell Row 4-Column 4 is "327".
These two cells represent the following facts:
Fact 1: The pdf "Cat Story" was downloaded 206 times on 15 Feb 2019.
Fact 2: The pdf "Cat Story" was downloaded 327 times on 16 Feb 2019.
In an SQL Server database you could store such facts in a simple table called "Downloads" with Column 1=Item name, Column 2=Date and Column 3 = Number of downloads.
My longwinded explanation is to make it clear that in your case you are using an Excel spreadsheet as a tool to display data whereas the purpose of a well-designed SQL Server database is to store facts without any duplications.
After you have decided on a normalised design for your SQL Server database, you can link your Excel spreadsheet directly to the data in your database.
I'm trying to develop a new reporting module for a resource management tool (PHP+Mysql).
I am trying to extract data in the following format from mysql:
I have a table that consists of date and location of multiple people(i.e Office, Home or Client).
Sample Data as in DB.
here date_plotted means the date at which the user is engaged and plotting_date represents when this particular entry was made in the system(the date). So User was plotted to be in office on 30th Oct and the same entry was made on 30th Oct.
Data as in resource table
The resource table represents the user table.
Any suggestions on how to do the same in mysql?
These are the primary tables which needs to be used.
The above table id done in excel for now to represent the outcome.
I'm new to SQL so haven't tried anything yet.
There is a tool for Windows that might simplify this operation. It's made by MySQL and called MySQL for Excel. In theory it should allow you to structure and make changes to MySQL databases as well as perform queries that result in spreadsheets.
Without knowing more about your data, for example being supplied an actual csv file to work with, and the parameters of the actual pull, whether it's fix dates always or if this is a dynamic pull based on a range this question could result in 100 different implementations that visually return similar results, but have massively different requirements overhead-wise in implementation.
Hello helpful internet strangers. I have created a fairly simple database for a client that has one main entry form and a search form that uses unbound text boxes for searching around 15 fields.
Details are below, but here is my problem: When I add new records and use the search form all the fields work as expected and return the correct results in the datasheet. When I sent the database to the client and they add new records, they are not returned in the search. I had them save the database with their records and send it back to me, and I confirmed that when I search for the records they added (I can see the records in the table) they are not returned in the search. If I add new records to the copy they sent me back, my records do appear in the search.
I created the database in Access 2013 on a Windows 8 machine. The client is using Access 2010 on Windows Vista.
The field types are text, dropdown and date. The dropdowns are all based on lookup queries so they store the id number in the main info table and pull the name value from the query. All of the fields in the search query are written to allow nulls, including the date range searches. And again, all fields test out correctly on my machine when I enter the records.
I went on site and compared settings and nothing jumped out at me except the different versions. I also watched the client enter new records and she didn't do anything 'wrong' or unusual. When I try to do a save as 2007-2010 it says I am using features that won't allow for that, but for the life of me I can't think of anything like that since this is really a very straightforward design.
I'm going to do a package as executable, but am highly doubtful that will help. Any insights?
Thanks in advance.
A few things. Most important - always, always develop in the earliest version of Access that the system will be used with. So, you need to rebuild, using Access 2010, period. Try creating a new blank database (in 2010) and importing the objects one (or a few) at a time. Make sure the system is split. One file for the FE (forms, queries, reports, code, etc.) and one for the BE (tables only). Make sure that the Filter On Load property of all forms is set to No. If the recordsource for the form contains a where clause, make sure that the newly entered records meet the criteria for the where clause.
While working with DB, I find useful using some tools, that help me to solve DB problems.
Some of them are:
1) Insert generator
2) A tool that can execute a script on a list of DB's
3) Finding a text in stored procedures and functions.
4) DB Back up scripts
My question is, what are most useful tools, scripts(anything else), that help you to work with SQL Server?
Thanks in advance.
UPDATE
I assume, there are no other tools for SQL Server 2008 or any other version?
Redgate has a collection of quite powerful tools for Sql Server.
Check out the SMSS Tools Pack.
I have stored procedures that do the following:
system utilities:
find and list every occurrence and info about a given column name or partial column name
find and list every occurrence and info about a given object name or partial object name
list out all the information for a given table, all columns, computed columns, column data types, nullability, defaults, identity, check constraints, index, pk, fk, triggers, and column comments.
find every trigger, view, stored procedure, or function that contains a given string
business utilities:
I also make stored procedures that work with business info. When working on an area of our application I'll make a procedure that displays out all the related info of a given thing. I'll usually display all the info using multiple PRINTs and SELECTs for everything that can join to the given PK (not if there are thousands or rows though). For example, one utiltiy would take a DoctorID as a parameter and list out all the doctor's info, offices that they work at, insurance they accept, etc. I like to include the table names in the output so I can remember where the data comes from without looking at the code. I also join in all the codes tables in these displays, so I'm not looking "A" but "Active (A)". After working on a system for a while, I have loads of these utilities, which greatly helps when a support call comes up or you need to debug a problem, etc. I usually build these as I develop, it is difficult to find time to go back and make them.