How adding ticket attachments with the Zendesk API using vba in Access [duplicate] - ms-access

Question

Overview
I have a Microsoft Word Add-In, written in VBA (Visual Basic for Applications), that compresses a document and all of it's related contents (embedded media) into a zip archive. After creating the zip archive it then turns the file into a byte array and posts it to an ASMX web service. This mostly works.
Issues
The main issue I have is transferring large files to the web site. I can successfully upload a file that is around 40MB, but not one that is 140MB (timeout/general failure).
A secondary issue is that building the byte array in the VBScript Word Add-In can fail by running out of memory on the client machine if the zip archive is too large.
Potential Solutions
I am considering the following options and am looking for feedback on either option or any other suggestions.
Option One
Opening a file stream on the client (MS Word VBA) and reading one "chunk" at a time and transmitting to ASMX web service which assembles the "chunks" into a file on the server.
This has the benefit of not adding any additional dependencies or components to the application, I would only be modifying existing functionality. (Fewer dependencies is better as this solution should work in a variety of server environments and be relatively easy to set up.)
Question:
Are there examples of doing this or any recommended techniques (either on the client in VBA or in the web service in C#/VB.NET)?
Option Two
I understand WCF may provide a solution to the issue of transferring large files by "chunking" or streaming data. However, I am not very familiar with WCF, and am not sure what exactly it is capable of or if I can communicate with a WCF service from VBA. This has the downside of adding another dependency (.NET 3.0). But if using WCF is definitely a better solution I may not mind taking that dependency.
Questions:
Does WCF reliably support large file transfers of this nature? If so, what does this involve? Any resources or examples?
Are you able to call a WCF service from VBA? Any examples?

Answer 1

I ended up implementing option one referenced in the original question.
I "chunk" the file in VBA and transfer each "chunk" to the web service. I based the VBA portion of the solution on the code found here: Copy Large File by Chunk with Progress Notification. Instead of copying to the file system, however, I send it to the server.
The Code: VBA Land
Here is the (fugly) VBA code that creates the file chunks:
Function CopyFileByChunk(fileName As String, sSource As String) As Boolean
Dim FileSize As Long, OddSize As Long, SoFar As Long
Dim Buffer() As Byte, f1 As Integer, ChunkSize As Long
On Error GoTo CopyFileByChunk_Error
f1 = FreeFile: Open sSource For Binary Access Read As #f1
FileSize = LOF(f1)
If FileSize = 0 Then GoTo Exit_CopyFileByChunk ' -- done!
ChunkSize = 5505024 '5.25MB
OddSize = FileSize Mod ChunkSize
Dim index As Integer
index = 0
If OddSize Then
ReDim Buffer(1 To OddSize)
Get #f1, , Buffer
index = index + 1
SoFar = OddSize
If UploadFileViaWebService(Buffer, fileName, index, SoFar = FileSize) Then
g_frmProgress.lblProgress = "Percent uploaded: " & Format(SoFar / FileSize, "0.0%")
Debug.Print SoFar, Format(SoFar / FileSize, "0.0%")
DoEvents
Else
GoTo CopyFileByChunk_Error
End If
End If
If ChunkSize Then
ReDim Buffer(1 To ChunkSize)
Do While SoFar < FileSize
Get #f1, , Buffer
index = index + 1
SoFar = SoFar + ChunkSize
If UploadFileViaWebService(Buffer, fileName, index, SoFar = FileSize) Then
g_frmProgress.lblProgress = "Percent uploaded: " & Format(SoFar / FileSize, "0.0%")
Debug.Print SoFar, Format(SoFar / FileSize, "0.0%")
DoEvents
Else
GoTo CopyFileByChunk_Error
End If
Loop
End If
CopyFileByChunk = True
Exit_CopyFileByChunk:
Close #f1
Exit Function
CopyFileByChunk_Error:
CopyFileByChunk = False
Resume Exit_CopyFileByChunk
End Function
Here is the referenced VBA method that uploads the chunks to the server:
Public Function UploadFileViaWebService(dataChunk() As Byte, fileName As String, index As Integer, lastChunk As Boolean) As Boolean
On Error GoTo ErrHand
Dim blnResult As Boolean
blnResult = False
'mdlConvert.SetProgressInfo "Connecting to the web server:" & vbNewLine & _
DQUOT & server_title() & DQUOT
If InternetAttemptConnect(0) = 0 Then
On Error Resume Next
Dim strSoapAction As String
Dim strXml As String
strXml = "<?xml version=""1.0"" encoding=""utf-8""?>" & _
"<soap:Envelope xmlns:xsi=""http://www.w3.org/2001/XMLSchema-instance"" xmlns:xsd=""http://www.w3.org/2001/XMLSchema"" xmlns:soap=""http://schemas.xmlsoap.org/soap/envelope/"">" & _
"<soap:Body>" & _
"<UploadZipFile xmlns=""http://something.com/"">" & _
"<zipBytes></zipBytes>" & _
"<index>" & index & "</index>" & _
"<isLastChunk>" & IIf(lastChunk, 1, 0) & "</isLastChunk>" & _
"</UploadZipFile>" & _
"</soap:Body>" & _
"</soap:Envelope>"
Dim objXmlhttp As Object
Dim objDom As Object
Set objXmlhttp = New MSXML2.xmlhttp
' Load XML
Set objDom = CreateObject("MSXML2.DOMDocument")
objDom.LoadXML strXml
'insert data chunk into XML doc
objDom.SelectSingleNode("//zipBytes").dataType = "bin.base64"
objDom.SelectSingleNode("//zipBytes").nodeTypedValue = dataChunk
' Open the webservice
objXmlhttp.Open "POST", webServiceUrl, False
' Create headings
strSoapAction = "http://something.com/UploadZipFile"
objXmlhttp.setRequestHeader "Content-Type", "text/xml; charset=utf-8"
objXmlhttp.setRequestHeader "SOAPAction", strSoapAction
' Send XML command
objXmlhttp.send objDom.XML
' Get all response text from webservice
Dim strRet
strRet = objXmlhttp.responseText
' Close object
Set objXmlhttp = Nothing
Set objDom = Nothing
'get the error if any
Set objDom = CreateObject("MSXML2.DOMDocument")
objDom.LoadXML strRet
Dim isSoapResponse As Boolean
isSoapResponse = Not (objDom.SelectSingleNode("//soap:Envelope") Is Nothing)
Dim error As String
If Not isSoapResponse Then
error = "Woops"
Else
error = objDom.SelectSingleNode("//soap:Envelope/soap:Body/soap:Fault/faultstring").text
End If
If error <> "" Then
ShowServerError error, True
blnResult = False
Else
Err.Clear 'clear the error caused in the XPath query above
blnResult = True
End If
'close dom object
Set objDom = Nothing
Else
GetErrorInfo "UploadFileViaWebService:InternetCheckConnection"
End If
ErrHand:
If Err.Number <> 0 Then
ShowError Err, "UploadFileViaWebService"
blnResult = False
End If
UploadFileViaWebService = blnResult
End Function
The Code: C# ASMX Web Service
Now, on the server side, the web service method accepts a few important parameters.
string fileName: The name of the
file (each chunk has the same file
name)
byte[] zipBytes: The contents
of each chunk
int index: The index
(used in conjunction with fileName
to provide unique ordered partial
files on the file system)
bool isLastChunk: This is the "i'm done -
go ahead and merge all the "chunks"
and clean up after yourself" flag.
int index and bool isLastChunk. With this context provided from the VBA world, I know enough to save each of these file chunks and then combine them when the isLastChunk flag is true.
/// <summary>
/// Accepts a chunk of a zip file. Once all chunks have been received, combines the chunks into a zip file that is processed.
/// </summary>
/// <param name="fileName">Name of the file.</param>
/// <param name="zipBytes">The collection of bytes in this chunk.</param>
/// <param name="index">The index of this chunk.</param>
/// <param name="isLastChunk">if set to <c>true</c> this is the last chunk.</param>
/// <returns>Whether the file was successfully read and parsed</returns>
/// <exception cref="ParserException">An error occurred while trying to upload your file. The details have been written to the system log.</exception>
[WebMethod]
public bool UploadZipFile(string fileName, byte[] zipBytes, int index, bool isLastChunk)
{
try
{
const string ModuleRootUrl = "/Somewhere/";
string folderName = HostingEnvironment.MapPath("~" + ModuleRootUrl);
string fullDirectoryName = Path.Combine(folderName, Path.GetFileNameWithoutExtension(fileName));
try
{
if (!Directory.Exists(fullDirectoryName))
{
Directory.CreateDirectory(fullDirectoryName);
}
string pathAndFileName = Path.Combine(fullDirectoryName, AddIndexToFileName(fileName, index));
using (var stream = new MemoryStream(zipBytes))
{
WriteStreamToFile(stream, pathAndFileName);
}
if (isLastChunk)
{
try
{
MergeFiles(fullDirectoryName, fileName, index);
// file transfer is done.
// extract the zip file
// and do whatever you need to do with its contents
// we'll assume that it works - but your "parsing" should return true or false
return true;
}
finally
{
DeleteDirectoryAndAllContents(fullDirectoryName);
}
}
}
catch
{
DeleteDirectoryAndAllContents(fullDirectoryName);
throw;
}
}
return false;
}
Here is the C# code that writes each incoming chunk the the file system:
/// <summary>
/// Writes the contents of the given <paramref name="stream"/> into a file at <paramref name="newFilePath"/>.
/// </summary>
/// <param name="stream">The stream to write to the given file</param>
/// <param name="newFilePath">The full path to the new file which should contain the contents of the <paramref name="stream"/></param>
public static void WriteStreamToFile(Stream stream, string newFilePath)
{
using (FileStream fs = File.OpenWrite(newFilePath))
{
const int BlockSize = 1024;
var buffer = new byte[BlockSize];
int numBytes;
while ((numBytes = stream.Read(buffer, 0, BlockSize)) > 0)
{
fs.Write(buffer, 0, numBytes);
}
}
}
Here is the C# code to merge all of the zip file "chunks":
/// <summary>
/// Merges each file chunk into one complete zip archive.
/// </summary>
/// <param name="directoryPath">The full path to the directory.</param>
/// <param name="fileName">Name of the file.</param>
/// <param name="finalChunkIndex">The index of the last file chunk.</param>
private static void MergeFiles(string directoryPath, string fileName, int finalChunkIndex)
{
var fullNewFilePath = Path.Combine(directoryPath, fileName);
using (var newFileStream = File.Create(fullNewFilePath))
{
for (int i = 1; i <= finalChunkIndex; i++)
{
using (var chunkFileStream = new FileStream(AddIndexToFileName(fullNewFilePath, i), FileMode.Open))
{
var buffer = new byte[chunkFileStream.Length];
chunkFileStream.Read(buffer, 0, (int)chunkFileStream.Length);
newFileStream.Write(buffer, 0, (int)chunkFileStream.Length);
}
}
}
}

Answer 2

I've transmitted large files like this using MTOM encoding.
More info on MTOM here: http://msdn.microsoft.com/en-us/library/aa395209.aspx
You can download an MTOM sample here: http://msdn.microsoft.com/en-us/library/ms751514.aspx
Check out Bustamante's book on WCF if you want more on MTOM.
As for the VBA call, I'm not an expert in that area therefore I don't have any info in regards to it.