I have 45 CSV files stores in google cloud storage in the same folder, when wild carding these into a data set table I am finding that some of the rows are missing once I connect the data to tableau. If I just select one of the files all the data will appear. all the files are called "PMPRO_PIVOT_ASDKE" the last 5 digits will change for each file. I have tried wildcarding with "PMPRO_PIVOT*" and it will take data from each file but some of the data is missing from each file.
Any ideas would be great as I've been trying to solve this all day.
Related
Using Azure Data Factory I want to achieve 2 similar things.
1-) Many files (csv or such) in a blob container under different folders, I want to take first line (which is header and in some cases remove multiple starting lines) from each file and concat all left from all files into a single file also in the blob
2-) Many json files (each containing multiple json but all files in the same folder), I also want to convert them to a single csv file (concat all csv version of the json files)
Then we will import that single file into a sql server or synapse table using bulk insert or openrowset or such. Import section we got it working.
How do we concat many files in different directories into one or similarly many json files after converting them to csv concat them.
Few addons
Assume 5 csv files are new, I will hit a sql server database and see if those files are imported already, lets say only 3 is not imported, sql server will return a resultset adding a unique integer fileid and the filename.
In the concat csv the first column is the fileid we get from the database, that column does not exists in the csv, similar concept for the json file, each json file contains multiple records and the fileid will be repeated for the record in the same json file during concat csv file creation
Also in the same blob in the root there are multiple folders, each folder for a certain file type. Within that folder many subfolders (multiple levels) created when new files are added.
When we ran the import process 30 minutes ago, we need a way to detect all new files added to the subfolder structure since the last import
This solution must be fast and efficient and it will be part of our ADF pipeline
I got a Problem, cause I'm totally new to sql and have to kinda learn it in an internship. So I had to import huge txt files into a database in phpmyadmin (took me for ever but managed it with load data infile). Now my task is to find a way to control if the data of the tables is the same as the data of the given txt files.
Is there any possibility to do so ?
Have you tried exporting the data through phpMyAdmin using a different file format instead of .sql? phpMyAdmin gives you several choices including CSV, OpenOffice spreadsheets. That would make your compare easier. You could use Excel, sort the data and you'd have a quicker compare.
the best way to do so is to load, and then extract.
Compare your extract with the original file.
Another way could be to count the number of line in both table and file. And extract few lines, and verify that they both exists. This is less precise.
But this has nothing to do with SQL. It is just a test logic.
I know how to import a text file into MySQL database by using the command
LOAD DATA LOCAL INFILE '/home/admin/Desktop/data.txt' INTO TABLE data
The above command will write the records of the file "data.txt" into the MySQL database table. My question is that I want to erase the records form the .txt file once it is stored in the database.
For Example: If there are 10 records and at current point of time 4 of them have been written into the database table, I require that in the data.txt file these 4 records get erased simultaneously. (In a way the text file acts as a "Queue".) How can I accomplish this? Can a java code be written? Or a scripting language is to be used?
Automating this is not too difficult, but it is also not trivial. You'll need something (a program, a script, ...) that can
Read the records from the original file,
Check if they were inserted, and, if they were not, copy them in another file
Rename or delete the original file, and rename the new file to replace the original one.
There might be better ways of achieving what you want to do, but, that's not something I can comment on without knowing your goal.
I have 2 separate sql databases, they both have the same field but are not attached and are completely separate files. One of the database files has a few hundred rows of data and I want to copy a few of those rows into the other database file. Some people have said to use sql statements to copy the data but the databases are not linked in any way so I am not sure as to how these statements would work. Is there no software where I can just select the correct rows and copy them over, or create a new database with the ones selected?
I hope this makes sense, thanks.
Regardless of the database platform you are using, there should be commands/tools that will allow you to perform bulk data imports and exports from/to a file (e.g. a CSV file). Try exporting the rows you wish to copy from the database on the first server into an intermediate file, copying that file to the second server, and then importing it into that database.
While uploading excel csv to mysql i find that '1000' becomes 1.00 and max value remains 999.99. While uploading same table with microsoft words spreadsheet it is fine but it has limitation of numbers of rows.how i can upload large table to mysql, i have csv data in .text files that has to be converted to .csv
It might be due to the thousand separator. Make sure that in your CSV, you see "1000" and not "1,000".
Also, see mySQL load data local infile incorrect number format, it might help you out.
The problem is either the formatting in your excel file or the limitations of numbers in your mysql table. Can you upload the file to google docs and show us the query which is used to create the table (alternatively - a screenshot of the table's structure)?