I have 40 csv files. All the files have different column names. I want a list of column names of each csv in a table format(in csv or in excel). The new file should contains list of column names from each csv file and corresponding csv file name.
I am doing it manually for now but if the number of files increases, it will become problematic. I want to do it using code.
Note: It can be a very trivial thing but I am not a techie so please help.
Related
I retrieved a data which has chemical compounds and related data in CSV. I am using SQL for sorting and retrieving the data that I want and I am very new to this.
The issue I am facing is that the compounds are named as compund_1, compound_2, and I want to rename them to their respective compound name( e.g.: compound_1 is Nicotine, compound_2 is Aspirin).
The issue is there are over 5500 of these and I'd like to know if it's possible for me to replace the names in bulk.
you can create a table with columns
required to store chemical compounds and related data.
import the csv file to this table.
create another csv file with compounds column and their names (asprin...).
import this csv file into the table. that will replace all compound_1 ... with their names.
I have the following problem I would like to solve.
I have a main table file that contains two columns: a unique identifier and a sequence of numerical values. The file looks something like this. The numerical values were copied into this file from a number of different Excel or .csv files (not more than 10). Each of these source files contain a part of the numerical values in the main file. These files look something like this.
I know the approximate, although not the exact length of each of the sub-sequences making up the numerical values in the column in the main file. I do not know the name of the .csv or Excel file where they originally came from.
I would like to do three things:
For each part of the values in the main file, determine which source file they originally came from.
Identify all numerical values that are in the same column but above and below this sub-sequence in the source file.
Copy these numerical values from the column of the sub-sequence in the source file into a new column in the main file.
Save. The results should look like this.
To restate this, the problems are:
I do not know the exact length of each sub-sequence, although I know the approximate length.
I do not know the source file for each sub-sequence of the numerical values in the main file.
In principle, this could be done by hand, but I would rather not. Does someone have an idea how to approach this?
We have a requirement to standardize the output of USQL.
The USQL reads the JSON (source file) data and convert it to csv format.
The problem is that the number of columns we have for each row in csv is not the same because of missing data in the source. Sometimes the result set of USQL have a row in csv with "N" columns, another row is with "N+1" columns (cells). We would like to standardize the output, having the same number columns in csv for all the rows.
How do we achieve this? We don't have any control over source file data, we would need to do the standardization while processing.
Has anyone faced similar challenges and found a solution?
I'm new to Tableau and now have a CSV file without column names. All data are stored in this CSV file. There is another CSV file contains all column names. I am trying to add these names to the unnamed data file. Is there a way to do it?
I know I could open the CSV file and copy the name manually, but I wish I could operate it in the Tableau.
Thanks.
You can connect to a CSV file that does not have a header row, and then name the fields yourself in Tableau when editing the data connection.
Click on the settings (gear shaped) icon on the right side of the table in the data source pane.
Then specify that there is no header row and that Tableau should generate names for the fields.
Then rename the fields from their generated names like F2, F2 etc to something meaningful, by clicking on the little black triangle icon next to the generated field names and choosing rename.
i trying to figure out if its possible to pre-process a CSV file in SSIS before importing the Data into SQL.
I currently receive a file that contains 8 tables with different structures in one flat file.
the Tables are identified by a row with the Table name in it encapsulated by Square Brackets i.e. [DOL_PROD]
the the data is underneath in standard CSV format. Headers first and then the data.
the tables are split by a blank line and the process repeats for the next 7 tables.
[DOL_CONSUME]
TP Ref,Item Code,Description,Qty,Serial,Consume_Ref
12345,abc,xxxxxxxxx,4,123456789,abc
[DOL_ENGPD]
TP Ref,EquipLoc,BackClyLoc,EngineerCom,Changed,NewName
is it posible to split it out into seperate CSV files? or Process it in a loop?
i would really like to be able to perform this all with SSIS automatically.
Kind Regards,
Adam
You can't do that by flat file source and connection manager alone.
There are two ways to achieve your goal:
You can use Script Component as source of the rows and to process the files, and then you'd do whatever you want with a file programatically.
The other way, is to read your flat file treating every row as a single column (i.e. without specifying delimiter), and then, via Data Flow Transformations, you'd be splitting rows, recognizing table names, splitting flows and so on.
I'd strongly advise you to use Script Component, even if you'd have to learn .NET first, because the second option will be a nightmare :). I'd use Flat File Source to extract lines from file as single column, and thet work it in Script Component, rather than reading a "raw" file directly.
Here's a resource that should get you started: http://furrukhbaig.wordpress.com/2012/02/28/processing-large-poorly-formatted-text-file-with-ssis-9/