I work at a company with three different departments and about 15 different GIS users (ESRI). Our work is in natural resources and we have hundreds of large orthophotos and lidar data that take up a lot of space.
Right now, the three different departments "share" all the GIS data, but they store them on different servers (and have for 15 years). So at this point, even though most departments have the same data, they are stored in different places, organized differently, have different naming conventions, and taking up 3 times the storage space (since everyone has different instances of the same data), etc.
I have been tasked to consolidate all the data into one, shared, organized folder. We want to have everything stored in this "master" folder and have the departments using this master folder going forward.
We plan to have separate folders for orthophotos, lidar files, vector data, etc. Within those folders, we have agreed that every file should be named with some variation of the following metadata: year, source, and geography. Everyone has their own way organizing their data, and have for 15 years. Some people want it organized by year_source_geography, some want it geography_year_source, some want it source_year_geography, etc.
QUESTION:
1) Lets say that we all agree to sort the master folder by year_source_geography: is there a way to create other folders with different naming conventions that will "shortcut" to the master data? The idea is that everyone can still have their files (shortcuts) organized the way they want to, but without creating duplicate files. For example: in addition to the master folder which is sorted by year_source_geography, can we have another folder that is sorted by geography_year_source that will "shortcut" to the master list?
2) Do you have any comments on how I'm organizing this data? I'm fairly new to GIS organization, so any suggestions or comments on how I should be organizing this data is welcomed.
Thanks!
I would implement DB rather than plain files.
Start from the following: http://geojson.org/
You can add your properties, such as year, filenames, paths, etc...
Even if your users still want the data in plain files, I still would manage the links and paths in NoSQL DB. This will provide you a great level of flexibility. Push the data into AWC or similar platform.
You can start testing with MongoDB free service http://www.mlab.com
Related
I have a system that produces CSV data on a regular event-driven basis (say, daily). Each event triggers the creation of a new folder and a fixed set of CSV files, each representing different types of data. For instance:
PlansDB.csv - data for plans of action
StepsDB.csv - descriptions of steps used by different plans
GroupsDB.csv - data on groups that can handle plans
RoomsDB.csv - data on places where a group can work on a plan
ResultsDB.csv - the records of results from steps of a plan
These have fields that identify the relationships between the different files, and I have no problems creating a data model for the CSVs in any given folder.
But how do I switch between folders? Once I have a working data model and some reports built off it, I'd like to view those reports on specific folders of data. How does that work? Can I switch easily to yesterday's folder, or last weeks, etc. with minimal effort (preferably just pointing to the folder).
The CSV files maintain the same names across folders which represent the types of the data they store. Can Power BI use that?
And can I run reports over multiple folders maintaining this data model? I know of the Folder merge capability, but my attempts at using it just merges all files as if they were the same type, whereas I would need each type merged separately.
You need to change the data source. To do this, from "Edit Queries" select "Data source settings":
Then click "Change Source..." button and select the new folder. After that Power BI Desktop will tell you to apply the changes and will reload the data from the new folder:
I'm considering storing a working directory (i.e. recursive/nested folder/files) into a mysql database.
The idea is to have a 'projects' table, and a table that contains all folders / files with their corresponding paths in the tree (varchar).
Querying a project should return folders and files as a list of String paths (+some meta-data) that I will use to build the tree at the client.
I should mention that the 'file' records in my working directory are meta-data that represent json files in a mongodb datastore (they describe webpages as complex nested json). File records will be queried more frequently than folders (they don't get queried at all), and will be linked to other tables. Files have a different meaning for my app than folders, which are merely important for my working directory.
My question is what would be the best option:
Store files and folders in separate tables
Store files and folders in the same table (the records will be near identical), and use a FK on a joined table with 2 records "file"/"folder"
Short answer: "classic" relational databases (including mysql) are not very good at this. Here is a good link: Managing hierarchical data in mySQL
On the other hand, Wordpress is an application that does many of the things you're trying to do - and it does it all in mySql (or equivalent RDBMS).
Look here, look especially at the "taxonomies" section.
Another approach might be to look at a graph database, like neo4J.
'Hope that helps ...
Background:
I am making a website where I want modular administrative rights for read/write/edit priviledges. My intent is to allow for any number of access level types, and to base it off of folder structure.
As an example, root admins would have read/write/edit for all site pages. Group A may have read/write/edit to all files in the path www.example.com/section1/ (including subfolders), Group B would have read/write/edit to all files in www.example.com/section2/, and so on.
I have considered two options to impliment this: create a MySQL database that would hold:
Group Name (reference name for the access group)
Read (list of folders the group can read separated by comma)
Write (list of folders the group can write new content to separated by comma)
Edit (list of folders the group can change already existing information separated by comma)
The other option I considered is creating a 'GroupAccess.txt' file somewhere and hand-jamming the information into that to reference.
Question: What are the advanatages of each of these systems? Specifically, what do I gain from putting admin access information in a database versus a text file, and vice versa? (i'm looking for information on potential speed issues, ease of maintainability, ease of editing/changing the information that will be stored)
Note: I'm not looking for a 'which is better', I want to know specific advantages so I can make a better informed decision on what's best for me.
The first thing that comes to mind is that the database would be more secure over a text file for the simple reason a text file can be read over the internet as most web servers serve .txt file by default, this would allow for users with restricted access and non-users of the site to see the whole structure of you site and in turn can make you more open to possible attacks on certain areas of your site.
Another benefit of using a database is that you can easily use a join to check is a user has access to some content in the database where as with a file you'll need to read the file get the permissions and the go build the SQL and get the data from the database.
Those are just two of the things that have stuck out from reading your question, hope it helps.
I'm running an eCommerce store and I am being provided with a daily data feed of info like Item Number, Price, Description, Inventory Status, etc.
Some notes:
I know the URL of the .xls file
I need to modify the Item Number on the .xls for all products to add two letters to the beginning
Price and Inventory Status on the website database need to be updated daily for each item, matched up by Item Number
If Item Number does not exist, new item is created with all information contained in the excel sheet
This all needs to be fully automated (this is the part I need the most assistance with)
I used to have a company that took care of this for $50/month, but I now have access to the data myself. My programming experience is limited to web languages (PHP, HTML, etc.) and some basic C++. A side question would be whether or not it's worth taking this responsibility upon myself or if I should continue working with a company who has systems in place to handle this already.
If you can get the CSV instead of the XLS, load it yourself into a new table, update what you need and then insert the rows into your production table.
If you're stuck with the XLS, find a library for PHP that will allow you to parse it and then write the records to the new table.
As for your second question, yes, it's absolutely worthwhile to cutout the thieves who are charging you $600/year for something that should take you an hour or two to write yourself.
Good luck.
There are two suggestions here. One involves using mysqlimport, the other TOAD. If you need more explanation of how to implement this, expand your question.
I'm writing an online project asset tracker but I'm new to MySQL. What would be the best way of tracking projects, users, and assets for something like this? I have 3 tables for assets, users, and projects. Users should own projects and assets. Assets could be members of multiple projects, and projects should be able to be seen by multiple users.
The first method I figured would be to have a mediumtext field on each project with the id for every asset that it's linked to. Each asset would also have a mediumtext that will have every project id it's linked to. This is a problem though, since I can't really do a search without having to parse the text to find out the projects/assets it's attached to.
Another solution without parsing would be to have separate tables for the linking information, so for instance there would be an asset table with the asset id, project id, and userid that it's part of, and if it gets assigned to another project or user, there would be another entry into that table. This solution, though, will have assets that have multiple entries.
Another way of doing it would be to have the site create a table whenever a project is created, and that will store the asset and user information. Since there might be thousands of projects, this will crowd up the database pretty quickly, and creating tables is heavier on MySQL than entries, as far as I know.
I'm leaning toward the second solution. Is there anybody who knows a better way?
Quote:
have a mediumtext field on each
project with the id for every asset
that it's linked to
This is the worst design... maybe ever! Read up on database relations. Take an emergency crash course. Look at some example databases; MS Access has some pretty decent templates you could examine.
What you describe looks like it could be modelled with these relations:
project --- inf:inf --- users
asset --- 1:1 --- users
asset --- inf:inf --- projects
The many-to-many relations would go in a separate table.