Import Unknown Field Order From Text - ms-access

Question

I've been assigned the task of importing about 180 csv files into an access 2007 database. These files have been put together over the years and will be put into 1 of 3 folders. I have not set up any data checks or restrictions to these tables (such as primary keys, validation rules, or relationships). That will be done once the data has been imported. The data contained in these files are from a survey which has changed over the years. This change has caused the fields to change. The order of them has changed or sometimes a field is there and sometimes it is not. I do have a list of all the fields possible though and what table each csv file should be imported to, and know that all these fields can be text.
Here is my problem: Not knowing what the order of the columns or if a column will exist, is it possible to run a function to import these text files into their relative tables by mapping each column in the text file to it's associated column in the access table?
Each text file has headers which is useful to see shat they actually are, but there is no text qualifier which can be very annoying when dealing with id codes consisting entirely of numbers. Below is what I've tried so far. It gets the file location from a function elsewhere, adds each filename in that location to a collection, then for each file in that collection it tries to import it into it's relative field.
'Get file names from the folder and store them in a collection
temp = Dir(location & "\*.*")
Do While temp <> ""
fileNames.Add temp
temp = Dir
Loop
'Go through each file in the collection and preccess it as needed
For Each temp2 In fileNames
If (temp2 Like "trip*") Then 'Import trip files
'Gets the data from a query 'DoCmd.RunSQL "SELECT * FROM [Text;FMT=Delimited;HDR=YES;IMEX=2;CharacterSet=437;DATABASE=" & location & "].[" & temp2 & "] As csv;"
DoCmd.TransferText acImportDelim, "Trips_Import", "tbl_Trips", location & "\" & temp2, -1
End If
If (temp2 Like "catch*") Then 'Import catch files
DoCmd.TransferText acImportDelim, "Catch_Import", "tbl_Catch", location & "\" & temp2, -1
End If
If (temp2 Like "size*") Then 'Import size files
DoCmd.TransferText acImportDelim, "Size_Import", "tbl_Size", location & "\" & temp2, -1
End If
Next temp2

Answer 1

You can create a SELECT * query for each CSV file and open the query as a recordset. Open another recordset for the destination table.
Then for each row in the CSV recordset, add a row to the destination recordset, loop through the CSV Fields collection, and add each CSV field value to the destination field with the same name.
This approach is independent of the order in which the fields appear in the CSV file. It also doesn't matter if the CSV file includes only a subset of the fields present in the destination table. As long as each CSV field also exists in the table, it should work (assuming compatible data types, the value satisfies validation rules/constraints, etc.).
Dim db As DAO.Database
Dim fld As DAO.Field
Dim rsDest As DAO.Recordset
Dim rsSrc As DAO.Recordset
Dim strSelect As String
Dim strTableName As String
Set db = CurrentDb
'Go through each file in the collection and preccess it as needed
For Each temp2 In fileNames
Select Case Left(temp2, 4)
Case "trip"
strTableName = "tbl_Trips"
Case "catc"
strTableName = "tbl_Catch"
Case "size"
strTableName = "tbl_Size"
Case Else
' what should happen here?
' this will trigger an error at OpenRecordset(strTableName) ...
strTableName = vbNullString
' figure out a better alternative
End Select
strSelect = "SELECT csv.* FROM " & _
"[Text;FMT=Delimited;HDR=YES;IMEX=2;CharacterSet=437;DATABASE=" & _
Location & "].[" & temp2 & "] As csv;"
Debug.Print strSelect
Set rsSrc = db.OpenRecordset(strSelect, dbOpenSnapshot)
Set rsDest = db.OpenRecordset(strTableName, dbOpenTable, dbAppendOnly)
With rsSrc
Do While Not .EOF
rsDest.AddNew
For Each fld In .Fields
rsDest.Fields(fld.Name).value = fld.value
Next
rsDest.Update
.MoveNext
Loop
.Close
End With
rsDest.Close
Next temp2
Note: This is a RBAR (row by agonizing row) approach, so the performance will be less than stellar. However, I presumed you will do this only once, so the performance hit will not be a deal-breaker. If you need a faster set-based approach instead, you can build and execute an "append query" for each CSV file. To do that, you would first need to get the CSV field names, and then build the appropriate INSERT INTO statement.