i need procedure in VBA to import data into access from csv excel file without some records,, as header and footer. Example,,, i have table in csv file, which contains some
sentence which not belong table date
A1 this is some sentence title.......
A2 title
A3.......
A7 DATA DATA DATA DATA DATA
A8 rows DATA DATA DATA DATA DATA
......
....
A256 DATA DATA DATA DATA
A257 this is some sentence
My Acess shoud contain only rows between A7 to A256. Does anyone knows procedure or whatever in VBA who solves my problems ?
thanks a lot
Edit
The easiest way to do it is to link the CSV-file into the Access database as a table. Then you can work on this table as if it was an ordinary access table, for instance by creating an appropriate query based on this table that returns exactly what you want.
You can link the table either manually or with VBA like this
DoCmd.TransferText TransferType:=acLinkDelim, TableName:="tblImport", _
FileName:="C:\MyData.csv", HasFieldNames:=true
Update
Dim db As DAO.Database
' Re-link the CSV Table
Set db = CurrentDb
On Error Resume Next: db.TableDefs.Delete "tblImport": On Error GoTo 0
db.TableDefs.Refresh
DoCmd.TransferText TransferType:=acLinkDelim, TableName:="tblImport", _
FileName:="C:\MyData.csv", HasFieldNames:=true
db.TableDefs.Refresh
' Perform the import
db.Execute "INSERT INTO someTable SELECT col1, col2, ... FROM tblImport " _
& "WHERE NOT F1 IN ('A1', 'A2', 'A3')"
db.Close: Set db = Nothing
Your file seems quite small (297 lines) so you can read and write them quite quickly. You refer to Excel CSV, which does not exists, and you show space delimited data in your example. Furthermore, Access is limited to 255 columns, and a CSV is not, so there is no guarantee this will work
Sub StripHeaderAndFooter()
Dim fs As Object ''FileSystemObject
Dim tsIn As Object, tsOut As Object ''TextStream
Dim sFileIn As String, sFileOut As String
Dim aryFile As Variant
sFileIn = "z:\docs\FileName.csv"
sFileOut = "z:\docs\FileOut.csv"
Set fs = CreateObject("Scripting.FileSystemObject")
Set tsIn = fs.OpenTextFile(sFileIn, 1) ''ForReading
sTmp = tsIn.ReadAll
Set tsOut = fs.CreateTextFile(sFileOut, True) ''Overwrite
aryFile = Split(sTmp, vbCrLf)
''Start at line 3 and end at last line -1
For i = 3 To UBound(aryFile) - 1
tsOut.WriteLine aryFile(i)
Next
tsOut.Close
DoCmd.TransferText acImportDelim, , "NewCSV", sFileOut, False
End Sub
Edit re various comments
It is possible to import a text file manually into MS Access and this will allow you to choose you own cell delimiters and text delimiters. You need to choose External data from the menu, select your file and step through the wizard.
About importing and linking data and database objects -- Applies to: Microsoft Office Access 2003
Introduction to importing and exporting data -- Applies to: Microsoft Access 2010
Once you get the import working using the wizards, you can save an import specification and use it for you next DoCmd.TransferText as outlined by #Olivier Jacot-Descombes. This will allow you to have non-standard delimiters such as semi colon and single-quoted text.
Related
My company has a vendor providing a JSON feed of data that I need to load into our MS Access database every two hours. I need to:
load the data from the feed,
parse the JSON into a usable format for Access, and then
insert it into the database.
I came across this question discussing a similar issue, but there's no good description there as to how to implement this in MS Access. Any help gratefully appreciated!
Using the VBA JSON library, you certainly can import JSON formatted files into MS Access. The idea is to consider JSON data as a collection of dictionaries and Visual Basic provides the collection and dictionary as data structures.
Below are the steps:
Build a table to match the structure of expected JSON data
On the VBA IDE side of MS Access, import the JsonConverter.bas (from link above) into a new module
Still in the IDE, under Tools / References, check off the VBA Reference: Microsoft Scripting Runtime
Include the following code that reads the JSON text file, parses it as a collection of dictionaries (with keys and valeus), and appends values iteratively into Access table. Place code behind an Access form or module (example uses a one nested level JSON file)
JSON
[
{
"col1": somenumber,
"col2": "somestring",
"col3": "somestring",
"col4": "somestring",
"col5": "somestring"
}
]
VBA Code
Private Function JSONImport()
Dim db As Database, qdef As Querydef
Dim FileNum As Integer
Dim DataLine As String, jsonStr As String, strSQL As String
Dim p As Object, element As Variant
Set db = CurrentDb
' READ FROM EXTERNAL FILE
FileNum = FreeFile()
Open "C:\Path\To\JsonFile.json" For Input As #FileNum
' PARSE FILE STRING
jsonStr = ""
While Not EOF(FileNum)
Line Input #FileNum, DataLine
jsonStr = jsonStr & DataLine & vbNewLine
Wend
Close #FileNum
Set p = ParseJson(jsonStr)
' ITERATE THROUGH DATA ROWS, APPENDING TO TABLE
For Each element In p
strSQL = "PARAMETERS [col1] Long, [col2] Text(255), [col3] Text(255), " _
& "[col4] Text(255), [col5] Text(255); " _
& "INSERT INTO TableName (col1, col2, col3, col4, col5) " _
& "VALUES([col1], [col2], [col3], [col4], [col5]);"
Set qdef = db.CreateQueryDef("", strSQL)
qdef!col1 = element("col1")
qdef!col2 = element("col2")
qdef!col3 = element("col3")
qdef!col4 = element("col4")
qdef!col5 = element("col5")
qdef.Execute
Next element
Set element = Nothing
Set p = Nothing
End Function
Json file handling in MS Access is easy. Just rename the .json extension to .txt and use the text import function with the delimiter set to (:) and the text delimiter to (").
One line of code... Happy coding!
I'm trying to import a CSV file that is created from a web form I developed. When the form submits it creates a record in my CSV with a multitude of customer information.
As per requirements I needed to put it into a CSV, and then separately have it import into an Access database for others to use (Two steps required for server security).
The way I'm trying to do it is with a simple form with a button on it inside Access, that simply says Import, that will pull an update of the CSV whenever the user needs it.
My error is confusing me as it's stating
"Field 'F1' doesn't exist in destination table 'Applications' "
I do not have a field in my CSV labeled F1, or even any record that contains 'F1', and there is no field named F1 in my access table Applications (obviously).
Here is my VB module code from Access
Option Compare Database
Sub ImportingCSV()
Function Import()
On Error GoTo Macro1_Err
DoCmd.TransferText acImportDelim, "", "Applications", "C:\Users\ALee\Documents\formTesting22.csv", False, ""
Import:
Exit Function
Macro1_Err:
MsgBox Error$
Resume Macro1_Exit
End Function
And here is my CSV file format (spaced out for your readability)
OPUCN#WVQNAJT4PD,
2017.05.03,
test,
v,
90545452929,
4062033985,
No,
VM#TEST.VMTEST,
10003937683827,
test,
test,
689 395 3967,
2048 2983999,
No,rle#don.ca,
111 e Streeth south,
12,
Temporary,
Commercial,
100,
200,
300,
208/120V,
Three-Phase,
Underground (UG),
Ganged Position*,
23,
"dsbsdhfbslhfbshfbsdhlfbgshdfgsfslfgljshgfljshgfljshgflsj"
The error is telling me that the field for the second phone number ("4062033985" in the CSV) doesn't have a field in the table Applications, but it does! "F1" in the CSV is Customer Mobile. When I import manually through Access's import wizard this works fine.
Hopefully someone can point me in the right direction, not familiar with VB script or macros in access.
Don't import the file.
Link the csv file as a table. Then create a query to read and convert (purify) the data.
Use this query as source for further processing of the date like appending data to other tables.
a CSV file is a spreadsheet... try...
DoCmd.TransferSpreadsheet acImport, acSpreadsheetTypeExcel12Xml,[YourDestinationTable],"C:\YourFileDirectoryPath, filename, and extension",true,[Spreadsheet name if multiple sheet names]
There are all kinds of ways to do this sort of thing. This is certainly the simplest method.
Private Sub Command0_Click()
DoCmd.TransferText acImportDelim, "", "Book1", "C:\your_path_here\Book1.csv", True, ""
End Sub
Let's say you want to import several CSV files, all of the same type, into the same table. Just run the script below.
Option Compare Database
Option Explicit
Private Sub Command0_Click()
DoImport
End Sub
Function DoImport()
Dim strPathFile As String
Dim strFile As String
Dim strPath As String
Dim strTable As String
Dim blnHasFieldNames As Boolean
' Change this next line to True if the first row in CSV worksheet
' has field names
blnHasFieldNames = True
' Replace C:\Documents\ with the real path to the folder that
' contains the CSV files
strPath = "C:\your_path_here\"
' Replace tablename with the real name of the table into which
' the data are to be imported
strFile = Dir(strPath & "*.csv")
Do While Len(strFile) > 0
strTable = Left(strFile, Len(strFile) - 4)
strPathFile = strPath & strFile
DoCmd.TransferText acImportDelim, , strTable, strPathFile, blnHasFieldNames
' Uncomment out the next code step if you want to delete the
' EXCEL file after it's been imported
' Kill strPathFile
strFile = Dir()
Loop
End Function
You can do all kinds of other thins too; navigate to a file using the msoFileDialogFilePicker; you can loop through record sets and load them, one by one, into your table. As Gustav suggested, you can link to your file (staging) and write records into a table (production). You should probably try all of these methods, and play around with
I've been assigned the task of importing about 180 csv files into an access 2007 database. These files have been put together over the years and will be put into 1 of 3 folders. I have not set up any data checks or restrictions to these tables (such as primary keys, validation rules, or relationships). That will be done once the data has been imported. The data contained in these files are from a survey which has changed over the years. This change has caused the fields to change. The order of them has changed or sometimes a field is there and sometimes it is not. I do have a list of all the fields possible though and what table each csv file should be imported to, and know that all these fields can be text.
Here is my problem: Not knowing what the order of the columns or if a column will exist, is it possible to run a function to import these text files into their relative tables by mapping each column in the text file to it's associated column in the access table?
Each text file has headers which is useful to see shat they actually are, but there is no text qualifier which can be very annoying when dealing with id codes consisting entirely of numbers. Below is what I've tried so far. It gets the file location from a function elsewhere, adds each filename in that location to a collection, then for each file in that collection it tries to import it into it's relative field.
'Get file names from the folder and store them in a collection
temp = Dir(location & "\*.*")
Do While temp <> ""
fileNames.Add temp
temp = Dir
Loop
'Go through each file in the collection and preccess it as needed
For Each temp2 In fileNames
If (temp2 Like "trip*") Then 'Import trip files
'Gets the data from a query 'DoCmd.RunSQL "SELECT * FROM [Text;FMT=Delimited;HDR=YES;IMEX=2;CharacterSet=437;DATABASE=" & location & "].[" & temp2 & "] As csv;"
DoCmd.TransferText acImportDelim, "Trips_Import", "tbl_Trips", location & "\" & temp2, -1
End If
If (temp2 Like "catch*") Then 'Import catch files
DoCmd.TransferText acImportDelim, "Catch_Import", "tbl_Catch", location & "\" & temp2, -1
End If
If (temp2 Like "size*") Then 'Import size files
DoCmd.TransferText acImportDelim, "Size_Import", "tbl_Size", location & "\" & temp2, -1
End If
Next temp2
You can create a SELECT * query for each CSV file and open the query as a recordset. Open another recordset for the destination table.
Then for each row in the CSV recordset, add a row to the destination recordset, loop through the CSV Fields collection, and add each CSV field value to the destination field with the same name.
This approach is independent of the order in which the fields appear in the CSV file. It also doesn't matter if the CSV file includes only a subset of the fields present in the destination table. As long as each CSV field also exists in the table, it should work (assuming compatible data types, the value satisfies validation rules/constraints, etc.).
Dim db As DAO.Database
Dim fld As DAO.Field
Dim rsDest As DAO.Recordset
Dim rsSrc As DAO.Recordset
Dim strSelect As String
Dim strTableName As String
Set db = CurrentDb
'Go through each file in the collection and preccess it as needed
For Each temp2 In fileNames
Select Case Left(temp2, 4)
Case "trip"
strTableName = "tbl_Trips"
Case "catc"
strTableName = "tbl_Catch"
Case "size"
strTableName = "tbl_Size"
Case Else
' what should happen here?
' this will trigger an error at OpenRecordset(strTableName) ...
strTableName = vbNullString
' figure out a better alternative
End Select
strSelect = "SELECT csv.* FROM " & _
"[Text;FMT=Delimited;HDR=YES;IMEX=2;CharacterSet=437;DATABASE=" & _
Location & "].[" & temp2 & "] As csv;"
Debug.Print strSelect
Set rsSrc = db.OpenRecordset(strSelect, dbOpenSnapshot)
Set rsDest = db.OpenRecordset(strTableName, dbOpenTable, dbAppendOnly)
With rsSrc
Do While Not .EOF
rsDest.AddNew
For Each fld In .Fields
rsDest.Fields(fld.Name).value = fld.value
Next
rsDest.Update
.MoveNext
Loop
.Close
End With
rsDest.Close
Next temp2
Note: This is a RBAR (row by agonizing row) approach, so the performance will be less than stellar. However, I presumed you will do this only once, so the performance hit will not be a deal-breaker. If you need a faster set-based approach instead, you can build and execute an "append query" for each CSV file. To do that, you would first need to get the CSV field names, and then build the appropriate INSERT INTO statement.
I have a CSV file with 550,000+ rows.
I need to import this data into Access, but when I try it throws an error that the file is too large (1.7GB).
Can you recommend a way to get this file into Access?
Try linking instead of importing ("get external data" -> "link table" in 2003), that leaves the data in the CSV-file and reads from the file directly and in-place. It doesn't limit size (at least not anywhere near 1.7 GB). It may limit some of your read/update operations, but it will at least get you started.
I'd either try the CSV ODBC connector, or otherwise import it first in a less limited database (MySQL, SQL Server) and import it from there.
It seems that some versions of access have a hard 2GB limit on MDB files so you might get into trouble with that anyway.
Good luck.
You can also use an ETL tool. Kettle is an open source one (http://kettle.pentaho.org/) and really quite easy to use. To import a file into a database requires a single transformation with 2 steps: CSV Text Input and Table Output.
why do you using access for huge files ? use sqlexpress or firebird instead
I remember that Access has some size limitation around 2 Go. Going to free SQLExpress (limited to 4 Go) or free MySQL (no size limitation) could be easier.
Another option would be to do away with the standard import functions and write your own. I have done this one time before when some specific logic needed to be applied to the data before import. The basic structure is……
Open then file
Get the first line
Loop through until the end of the line
If we find a comma then move onto the next field
Put record into database
Get the next line repeat etc
I wrapped it up into a transaction that committed every 100 rows as I found that improved performance in my case but it would depend on your data if that helped.
However I would say that linking the data as others have said is the best solution, this is just an option if you absolutely have to have the data in access
Access creates a lot of overhead so even relatively small data sets can bloat the file to 2GB, and then it will shut down. Here are a couple of straightforward ways of doing the import. I didn't test this on huge files, but these concepts will definitely work on regular files.
Import data from a closed workbook (ADO)
If you want to import a lot of data from a closed workbook you can do this with ADO and the macro below. If you want to retrieve data from another worksheet than the first worksheet in the closed workbook, you have to refer to a user defined named range. The macro below can be used like this (in Excel 2000 or later):
GetDataFromClosedWorkbook "C:\FolderName\WorkbookName.xls", "A1:B21", ActiveCell, False
GetDataFromClosedWorkbook "C:\FolderName\WorkbookName.xls", "MyDataRange", Range ("B3"), True
Sub GetDataFromClosedWorkbook(SourceFile As String, SourceRange As String, _
TargetRange As Range, IncludeFieldNames As Boolean)
' requires a reference to the Microsoft ActiveX Data Objects library
' if SourceRange is a range reference:
' this will return data from the first worksheet in SourceFile
' if SourceRange is a defined name reference:
' this will return data from any worksheet in SourceFile
' SourceRange must include the range headers
'
Dim dbConnection As ADODB.Connection, rs As ADODB.Recordset
Dim dbConnectionString As String
Dim TargetCell As Range, i As Integer
dbConnectionString = "DRIVER={Microsoft Excel Driver (*.xls)};" & _
"ReadOnly=1;DBQ=" & SourceFile
Set dbConnection = New ADODB.Connection
On Error GoTo InvalidInput
dbConnection.Open dbConnectionString ' open the database connection
Set rs = dbConnection.Execute("[" & SourceRange & "]")
Set TargetCell = TargetRange.Cells(1, 1)
If IncludeFieldNames Then
For i = 0 To rs.Fields.Count - 1
TargetCell.Offset(0, i).Formula = rs.Fields(i).Name
Next i
Set TargetCell = TargetCell.Offset(1, 0)
End If
TargetCell.CopyFromRecordset rs
rs.Close
dbConnection.Close ' close the database connection
Set TargetCell = Nothing
Set rs = Nothing
Set dbConnection = Nothing
On Error GoTo 0
Exit Sub
InvalidInput:
MsgBox "The source file or source range is invalid!", _
vbExclamation, "Get data from closed workbook"
End Sub
Another method that doesn't use the CopyFromRecordSet-method
With the macro below you can perform the import and have better control over the results returned from the RecordSet.
Sub TestReadDataFromWorkbook()
' fills data from a closed workbook in at the active cell
Dim tArray As Variant, r As Long, c As Long
tArray = ReadDataFromWorkbook("C:\FolderName\SourceWbName.xls", "A1:B21")
' without using the transpose function
For r = LBound(tArray, 2) To UBound(tArray, 2)
For c = LBound(tArray, 1) To UBound(tArray, 1)
ActiveCell.Offset(r, c).Formula = tArray(c, r)
Next c
Next r
' using the transpose function (has limitations)
' tArray = Application.WorksheetFunction.Transpose(tArray)
' For r = LBound(tArray, 1) To UBound(tArray, 1)
' For c = LBound(tArray, 2) To UBound(tArray, 2)
' ActiveCell.Offset(r - 1, c - 1).Formula = tArray(r, c)
' Next c
' Next r
End Sub
Private Function ReadDataFromWorkbook(SourceFile As String, SourceRange As String) As Variant
' requires a reference to the Microsoft ActiveX Data Objects library
' if SourceRange is a range reference:
' this function can only return data from the first worksheet in SourceFile
' if SourceRange is a defined name reference:
' this function can return data from any worksheet in SourceFile
' SourceRange must include the range headers
' examples:
' varRecordSetData = ReadDataFromWorkbook("C:\FolderName\SourceWbName.xls", "A1:A21")
' varRecordSetData = ReadDataFromWorkbook("C:\FolderName\SourceWbName.xls", "A1:B21")
' varRecordSetData = ReadDataFromWorkbook("C:\FolderName\SourceWbName.xls", "DefinedRangeName")
Dim dbConnection As ADODB.Connection, rs As ADODB.Recordset
Dim dbConnectionString As String
dbConnectionString = "DRIVER={Microsoft Excel Driver (*.xls)};ReadOnly=1;DBQ=" & SourceFile
Set dbConnection = New ADODB.Connection
On Error GoTo InvalidInput
dbConnection.Open dbConnectionString ' open the database connection
Set rs = dbConnection.Execute("[" & SourceRange & "]")
On Error GoTo 0
ReadDataFromWorkbook = rs.GetRows ' returns a two dim array with all records in rs
rs.Close
dbConnection.Close ' close the database connection
Set rs = Nothing
Set dbConnection = Nothing
On Error GoTo 0
Exit Function
InvalidInput:
MsgBox "The source file or source range is invalid!", vbExclamation, "Get data from closed workbook"
Set rs = Nothing
Set dbConnection = Nothing
End Function
For really large files, you can try something like this . . .
INSERT INTO [Table] (Column1, Column2)
SELECT *
FROM [Excel 12.0 Xml;HDR=No;Database=C:\your_path\excel.xlsx].[SHEET1$];
OR
SELECT * INTO [NewTable]
FROM [Excel 12.0 Xml;HDR=No;Database=C:\your_path\excel.xlsx].[SHEET1$];
I have an access database, with a query made. I need to automate it so that each night this query can run and export to a tab delimited csv file. It is not possible to export a query to a csv file from within access. My question is, are there any tools that can select certain tables, or perform an sql query on an mdb file, and export to a csv file?
Actually, you can export a query to a csv file from within Access.
You can do this with a Macro using the TransferText method.
Macro:
Name = ExportQuery
Action = TransferText
Transfer Type = Export Delimited
Table Name = [name of your Access query]
File Name = [path of output file]
Has Field Names = [Yes or No, as desired]
You can execute the macro from the command line like this:
"[your MS Office path]\msaccess.exe" [your databse].mdb /excl /X ExportQuery /runtime
Since you're having trouble with TransferText in a macro try this:
1) Create a Module named "ExportQuery". In this module, create a function called "ExportQuery":
Function ExportQuery()
DoCmd.TransferText acExportDelim, , "[your query]", "[output file].csv"
End Function
2) Create a Macro named RunExportQuery:
Action = RunCode
Function Name = ExportQuery ()
VBScript works quite well with the Jet engine. However, I do not see why you say " It is not possible to export a query to a csv file from within access."
Sub TransferCSV()
DoCmd.TransferText acExportDelim, , "PutNameOfQueryHere", "C:\PutPathAnd\FilenameHere.csv", True
End Sub
Is the usual way in VBA.
EDIT:
It is possible to run a VBScript file (.vbs) from the command line. Here is some sample VBScript to output a tab delimited file.
db = "C:\Docs\LTD.mdb"
TextExportFile = "C:\Docs\Exp.txt"
Set cn = CreateObject("ADODB.Connection")
Set rs = CreateObject("ADODB.Recordset")
cn.Open _
"Provider = Microsoft.Jet.OLEDB.4.0; " & _
"Data Source =" & db
strSQL = "SELECT * FROM tblMembers"
rs.Open strSQL, cn, 3, 3
Set fs = CreateObject("Scripting.FileSystemObject")
Set f = fs.CreateTextFile(TextExportFile, True)
a = rs.GetString
f.WriteLine a
f.Close
SQL Server Integration Services is able to do the transformation that you are talking about. Don't be fooled by the name, because you don't need SQL Server in order to automate and run the packages.
http://msdn.microsoft.com/en-us/library/ms141026.aspx