My company has a vendor providing a JSON feed of data that I need to load into our MS Access database every two hours. I need to:
load the data from the feed,
parse the JSON into a usable format for Access, and then
insert it into the database.
I came across this question discussing a similar issue, but there's no good description there as to how to implement this in MS Access. Any help gratefully appreciated!
Using the VBA JSON library, you certainly can import JSON formatted files into MS Access. The idea is to consider JSON data as a collection of dictionaries and Visual Basic provides the collection and dictionary as data structures.
Below are the steps:
Build a table to match the structure of expected JSON data
On the VBA IDE side of MS Access, import the JsonConverter.bas (from link above) into a new module
Still in the IDE, under Tools / References, check off the VBA Reference: Microsoft Scripting Runtime
Include the following code that reads the JSON text file, parses it as a collection of dictionaries (with keys and valeus), and appends values iteratively into Access table. Place code behind an Access form or module (example uses a one nested level JSON file)
JSON
[
{
"col1": somenumber,
"col2": "somestring",
"col3": "somestring",
"col4": "somestring",
"col5": "somestring"
}
]
VBA Code
Private Function JSONImport()
Dim db As Database, qdef As Querydef
Dim FileNum As Integer
Dim DataLine As String, jsonStr As String, strSQL As String
Dim p As Object, element As Variant
Set db = CurrentDb
' READ FROM EXTERNAL FILE
FileNum = FreeFile()
Open "C:\Path\To\JsonFile.json" For Input As #FileNum
' PARSE FILE STRING
jsonStr = ""
While Not EOF(FileNum)
Line Input #FileNum, DataLine
jsonStr = jsonStr & DataLine & vbNewLine
Wend
Close #FileNum
Set p = ParseJson(jsonStr)
' ITERATE THROUGH DATA ROWS, APPENDING TO TABLE
For Each element In p
strSQL = "PARAMETERS [col1] Long, [col2] Text(255), [col3] Text(255), " _
& "[col4] Text(255), [col5] Text(255); " _
& "INSERT INTO TableName (col1, col2, col3, col4, col5) " _
& "VALUES([col1], [col2], [col3], [col4], [col5]);"
Set qdef = db.CreateQueryDef("", strSQL)
qdef!col1 = element("col1")
qdef!col2 = element("col2")
qdef!col3 = element("col3")
qdef!col4 = element("col4")
qdef!col5 = element("col5")
qdef.Execute
Next element
Set element = Nothing
Set p = Nothing
End Function
Json file handling in MS Access is easy. Just rename the .json extension to .txt and use the text import function with the delimiter set to (:) and the text delimiter to (").
One line of code... Happy coding!
Related
I have an excel cell value [[{"Name":Ashwin ,"Age":64}],[],[{"Name":Shakur ,"Age":64,"Gender":Male}]]
I need to display the value of gender in cells.
Please find below my code:
Option Explicit
Sub ExampleSplit()
Dim s As String, vx() As String
My_array = Worksheets("sheet1").Cells(1, 1)
vx = Split(My_array, "{")
Array_need = "{" & Split(vx(UBound(vx)), "}")(0) & "}"
Set Jsonobject = JsonConverter.ParseJson(Array_need)
For Each Item In Jsonobject
If Item = "Gender" Then
Worksheets("sheet1").Cells(1, 2) = Item("Gender")
End If
Next
End Sub
After running sucessfuly,value "Male" should be in worksheets("sheet1").cells(1,2).But for me it was throwing "type mismatch"
I wrote PrintJSONAccessors() to answer a similar question: Using VBA and VBA-JSON to access JSON data from Wordpress API. My sub routine prints the proper way to access the json data to the Immediate Window.
The sample code is not valid JSON. It is missing double quotes around its string values.
[[{"Name":Ashwin ,"Age":64}],[],[{"Name":Shakur ,"Age":64,"Gender":Male}]]
This is the valid version:
[[{"Name":"Ashwin" ,"Age":64}],[],[{"Name":"Shakur" ,"Age":64,"Gender":"Male"}]]
Here is how I prepare to extract the JSON data:
Sub Prep()
Dim Data As Variant
Data = Worksheets("sheet1").Cells(1, 1).Value
Set Data = JsonConverter.ParseJson(Data)
PrintJSONAccessors Data, "Data"
Stop
End Sub
I put the Stop in the code so that I can test output in the Immediate Window.
Notice the data is a Dictionary inside a Collection inside another Collection.
I am trying to pull JSON values from a URL that I am working with at the moment. I may have done something like this before but I dont know what I'm missing here.
Here is the URL - https://eu-offering.kambicdn.org/offering/v2018/888/listView/golf.json?lang=en_GB&market=GB&client_id=2&channel_id=1&ncid=1568916879040&useCombined=true
And an image for clarity of what is needed to be extracted.
I ran a test using Tinman's approach as can be found here - How to get, JSON values to Work in VBA-JSON? , but i can't even apply his function, PrintJSONAccessors(), here
Public Sub exceljson()
Dim http As Object
Set http = CreateObject("MSXML2.XMLHTTP")
http.Open "GET",
"https://eu-offering.kambicdn.org/offering/v2018/888/listView/golf.json?lang=en_GB&market=GB&client_id=2&channel_id=1&ncid=1568916879040&useCombined=true", False
http.Send
Dim results As Variant
results = BitfinexTextToArray(http.responseText)
Worksheets(1).Range("A1").Resize(UBound(results), UBound(results,2)).Value = results
MsgBox ("complete")
End Sub
Function BitfinexTextToArray(responseText As String) As Variant
Dim item As Variant, JSON As Object
Dim MaxColumns As Long
Set JSON = ParseJson(responseText)
For Each item In JSON
If item.Count > MaxColumns Then MaxColumns = item.Count
Next
Dim results As Variant
ReDim results(1 To JSON.Count, 1 To MaxColumns)
Dim c As Long, r As Long
For Each item In JSON
r = r + 1
For c = 1 To item.Count
results(r, c) = item(c)
Next
Next
BitfinexTextToArray = results
End Function
I need help with pulling the following item values from each of the JSON "event"
1. "englishName"
2. "participant"
3. "oddsFractional"
NOTE: my example uses the JsonConverter library and requires you to add a reference to the Microsoft Scripting Runtime to access the Dictionary object.
I set up a test file with JSON loaded from your URL above. After parsing the JSON data, the exercise becomes understanding how the various levels are nested and what type of data structure is being used. In your JSON, it's a mix of Collection, Array, and Dictionary in various combinations. My example below shows how you have to stack up these nested references to get the data you're looking for.
Review the information in this answer to understand how the JSON is parsed into a hierarchical data structure.
Option Explicit
Public Sub test()
Dim fileNum As Long
fileNum = FreeFile()
Dim filename As String
filename = "C:\Temp\testdata.json"
Dim jsonInput As String
Open filename For Input As #fileNum
jsonInput = Input$(LOF(fileNum), fileNum)
Close fileNum
Dim json As Object
Set json = ParseJson(jsonInput)
Debug.Print " English Name = " & json("events")(1)("event")("englishName")
Debug.Print " Participant = " & json("events")(1)("betOffers")(1)("outcomes")(2)("participant")
Debug.Print "Odds Fractional = " & json("events")(1)("betOffers")(1)("outcomes")(2)("oddsFractional")
End Sub
An even better solution will be to create an intermediate variable and then loop over the contents in an array (or collection or dictionary).
Background Information:
I am not very savvy with VBA, or Access for that matter, but I have a VBA script that creates a file (a KML to be specific, but this won't matter much for my question) on the users computer and writes to it using variables that link to records in the database. As such:
Dim MyDB As Database
Dim MyRS As Recordset
Dim QryOrTblDef As String
Dim TestFile As Integer
QryOrTblDef = "Table1"
Set MyDB = CurrentDb
Set MyRS = MyDB.OpenRecordset(QryOrTblDef)
TestFile = FreeFile
Open "C:\Testing.txt"
Print #TestFile, "Generic Stuff"
Print #TestFile, MyRS.Fields(0)
etc.
My Situation:
I have a very large string(a text document with a large list of polygon vertex coordinates) that I want to add to a variable to be printed to another file (a KML file, noted in the above example). I was hoping to add this text file containing coordinates as an attachment datatype to the Access database and copy its contents into a variable to be used in the above script.
My Question:
Is there a way I can access and copy the data from an attached text file (attached as an attachment data type within a field of an MS Access database) into a variable so that I can use it in a VBA script?
What I have found:
I am having trouble finidng information on this topic I think mainly because I do not have the knowledge of what keywords to be searching for, but I was able to find someones code on a forum, "ozgrid", that seems to be close to what I want to do. Though it is just pulling from a text file on disk rather than one attached to the database.
Code from above mentioned forum that creates a function to access data in a text file:
Sub Test()
Dim strText As String
strText = GetFileContent("C:\temp\x.txt")
MsgBox strText
End Sub
Function GetFileContent(Name As String) As String
Dim intUnit As Integer
On Error Goto ErrGetFileContent
intUnit = FreeFile
Open Name For Input As intUnit
GetFileContent = Input(LOF(intUnit), intUnit)
ErrGetFileContent:
Close intUnit
Exit Function
End Function
Any help here is appreciated. Thanks.
I am a little puzzled as to why a memo data type does not suit if you are storing pure text, or even a table for organized text. That being said, one way is to output to disk and read into a string.
''Ref: Windows Script Host Object Model
Dim fs As New FileSystemObject
Dim ts As TextStream
Dim rs As DAO.Recordset, rsA As DAO.Recordset
Dim sFilePath As String
Dim sFileText As String
sFilePath = "z:\docs\"
Set rs = CurrentDb.OpenRecordset("maintable")
Set rsA = rs.Fields("aAttachment").Value
''File exists
If Not fs.FileExists(sFilePath & rsA.Fields("FileName").Value) Then
''It will save with the existing FileName, but you can assign a new name
rsA.Fields("FileData").SaveToFile sFilePath
End If
Set ts = fs.OpenTextFile(sFilePath _
& rsA.Fields("FileName").Value, ForReading)
sFileText = ts.ReadAll
See also: http://msdn.microsoft.com/en-us/library/office/ff835669.aspx
i need procedure in VBA to import data into access from csv excel file without some records,, as header and footer. Example,,, i have table in csv file, which contains some
sentence which not belong table date
A1 this is some sentence title.......
A2 title
A3.......
A7 DATA DATA DATA DATA DATA
A8 rows DATA DATA DATA DATA DATA
......
....
A256 DATA DATA DATA DATA
A257 this is some sentence
My Acess shoud contain only rows between A7 to A256. Does anyone knows procedure or whatever in VBA who solves my problems ?
thanks a lot
Edit
The easiest way to do it is to link the CSV-file into the Access database as a table. Then you can work on this table as if it was an ordinary access table, for instance by creating an appropriate query based on this table that returns exactly what you want.
You can link the table either manually or with VBA like this
DoCmd.TransferText TransferType:=acLinkDelim, TableName:="tblImport", _
FileName:="C:\MyData.csv", HasFieldNames:=true
Update
Dim db As DAO.Database
' Re-link the CSV Table
Set db = CurrentDb
On Error Resume Next: db.TableDefs.Delete "tblImport": On Error GoTo 0
db.TableDefs.Refresh
DoCmd.TransferText TransferType:=acLinkDelim, TableName:="tblImport", _
FileName:="C:\MyData.csv", HasFieldNames:=true
db.TableDefs.Refresh
' Perform the import
db.Execute "INSERT INTO someTable SELECT col1, col2, ... FROM tblImport " _
& "WHERE NOT F1 IN ('A1', 'A2', 'A3')"
db.Close: Set db = Nothing
Your file seems quite small (297 lines) so you can read and write them quite quickly. You refer to Excel CSV, which does not exists, and you show space delimited data in your example. Furthermore, Access is limited to 255 columns, and a CSV is not, so there is no guarantee this will work
Sub StripHeaderAndFooter()
Dim fs As Object ''FileSystemObject
Dim tsIn As Object, tsOut As Object ''TextStream
Dim sFileIn As String, sFileOut As String
Dim aryFile As Variant
sFileIn = "z:\docs\FileName.csv"
sFileOut = "z:\docs\FileOut.csv"
Set fs = CreateObject("Scripting.FileSystemObject")
Set tsIn = fs.OpenTextFile(sFileIn, 1) ''ForReading
sTmp = tsIn.ReadAll
Set tsOut = fs.CreateTextFile(sFileOut, True) ''Overwrite
aryFile = Split(sTmp, vbCrLf)
''Start at line 3 and end at last line -1
For i = 3 To UBound(aryFile) - 1
tsOut.WriteLine aryFile(i)
Next
tsOut.Close
DoCmd.TransferText acImportDelim, , "NewCSV", sFileOut, False
End Sub
Edit re various comments
It is possible to import a text file manually into MS Access and this will allow you to choose you own cell delimiters and text delimiters. You need to choose External data from the menu, select your file and step through the wizard.
About importing and linking data and database objects -- Applies to: Microsoft Office Access 2003
Introduction to importing and exporting data -- Applies to: Microsoft Access 2010
Once you get the import working using the wizards, you can save an import specification and use it for you next DoCmd.TransferText as outlined by #Olivier Jacot-Descombes. This will allow you to have non-standard delimiters such as semi colon and single-quoted text.
I'm busy with an applicaton which reads space delimited log files ranging from 5mb to 1gb+ in size, then stores this information to a MySQL database for later use when printing reports based upon the information contained in the files. The methods I've tried / found work but are very slow.
Am I doing something wrong? or is there a better way to handle very large text files?
I've tried using textfieldparser as follows:
Using parser As New TextFieldParser("C:\logfiles\testfile.txt")
parser.TextFieldType = FieldType.Delimited
parser.CommentTokens = New String() {"#"}
parser.Delimiters = New String() {" "}
parser.HasFieldsEnclosedInQuotes = False
parser.TrimWhiteSpace = True
While Not parser.EndOfData
Dim input As String() = parser.ReadFields()
If input.Length = 10 Then
'add this to a datatable
End If
End While
End Using
This works but is very slow for the larger files.
I then tried using an OleDB connection to the text file as per the following function in conjunction with a schema.ini file I write to the directory beforehand:
Function GetSquidData(ByVal logfile_path As String) As System.Data.DataTable
Dim myData As New DataSet
Dim strFilePath As String = ""
If logfile_path.EndsWith("\") Then
strFilePath = logfile_path
Else
strFilePath = logfile_path & "\"
End If
Dim mySelectQry As String = "SELECT * FROM testfile.txt WHERE Client_IP <> """""
Dim myConnection As New System.Data.OleDb.OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & strFilePath & ";Extended Properties=""text;HDR=NO;""")
Dim dsCmd As New System.Data.OleDb.OleDbDataAdapter(mySelectQry, myConnection)
dsCmd.Fill(myData, "logdata")
If Not myConnection.State = ConnectionState.Closed Then
myConnection.Close()
End If
Return myData.Tables("logdata")
End Function
The schema.ini file:
[testfile.txt]
Format=Delimited( )
ColNameHeader=False
Col1=Timestamp text
Col2=Elapsed text
Col3=Client_IP text
Col4=Action_Code text
Col5=Size double
Col6=Method text
Col7=URI text
Col8=Ident text
Col9=Hierarchy_From text
Col10=Content text
Anyone have any ideas how to read these files faster?
-edit-
Corrected a typo in the code above
There are two potentially slow operations there:
File reading
Inserting lots of data into the db
Separate them and test which is taking the most time. I.e. write one test program that simply reads the file, and another test program that just inserts loads of records. See which one is slowest.
One problem could be that you are reading the whole file into memory?
Try reading it line by line with a Stream. Here is a code example copied from MSDN
Imports System
Imports System.IO
Class Test
Public Shared Sub Main()
Try
' Create an instance of StreamReader to read from a file.
' The using statement also closes the StreamReader.
Using sr As New StreamReader("TestFile.txt")
Dim line As String
' Read and display lines from the file until the end of
' the file is reached.
Do
line = sr.ReadLine()
If Not (line Is Nothing) Then
Console.WriteLine(line)
End If
Loop Until line Is Nothing
End Using
Catch e As Exception
' Let the user know what went wrong.
Console.WriteLine("The file could not be read:")
Console.WriteLine(e.Message)
End Try
End Sub
End Class
From the top of my head id say try to impelement some kind of threading to spread the workload.