MySQL Connector/Net Bulk Uploader not finding file on remote server - mysql

I have pored through many pages trying to find an answer but have had no luck. I have a .NET page built in C# that has been working fine until a few days ago. Now it isn't working and I'm pulling my hair out to find out why.
The page has a file uploader that uploads a .csv file and saves it to a folder on the web server. Then it uses the MySQL Bulk Uploader to insert the records into the database on another server.
I have confirmed the file is uploading to the correct folder, but when MySQL tries to insert the records, it fails with the message "File 'E:\inetpub\wwwroot\training\data_uploads\filename.csv' not found (Errcode: 2 - No such file or directory)"
This page has worked for several years without any problem, but I updated some of the NuGet packages and removed some that were not being used, and now it's stopped working. What am I missing? Is there a package or a .dll I need to add back in? Unfortunately, I don't remember what I removed.
Here's the code I'm using:
protected void btnGo_Click(object sender, EventArgs e)
{
try
{
//if file is selected for upload
if (btnSelectFile.HasFile)
{
//upload data file to server
string path = string.Concat(Server.MapPath("~/data_uploads/" + btnSelectFile.FileName));
btnSelectFile.SaveAs(path);
string conString = ConfigurationManager.ConnectionStrings["nameOfConnectionString"].ConnectionString;
MySqlConnection conn = new MySqlConnection(conString);
conn.Open();
//get rid of old data
MySqlCommand truncateTerms = new MySqlCommand("TRUNCATE terms_temp;", conn);
truncateTerms.ExecuteNonQuery();
//create bulk uploader and set parameters
var bl = new MySqlBulkLoader(conn);
bl.TableName = "terms_temp";
bl.FieldTerminator = ",";
bl.FieldQuotationCharacter = '"';
bl.LineTerminator = "\r\n";
bl.FileName = path;
bl.NumberOfLinesToSkip = 2;
//insert data
var inserted = bl.Load(); //This is where it fails
conn.Close();
//do some other stuff
catch (Exception ex)
{
Label1.ForeColor = System.Drawing.Color.Red;
Label1.Text = ex.Message.ToString();
}
}

If you're bulk-loading a file that's stored on the web server, not the database server, you need to set MySqlBulkLoader.Local = true, to indicate that the file is local to the database client. Otherwise, the server will give an error that the file isn't found.
For security reasons you will also need to set AllowLoadLocalInfile=true in your connection string to enable this feature.

Related

How to send a file greater than 25 MB using Java Mail API

I am designing an application which sends Email with attachments using Gmail's smtp host. But when the file is larger than 25 MB, then I get an error saying that "552-5.2.3 Your message exceeded Google's message size limits. Please visit https://support.google.com/mail/?p=MaxSizeError to view our size guidelines.
188sm2692677pfg.11 -gsmtp"
final String username = "username#gmail.com";
final String password = "password";
Properties prop = new Properties();
prop.put("mail.smtp.host", "smtp.gmail.com");
prop.put("mail.smtp.port", "587");
prop.put("mail.smtp.auth", "true");
prop.put("mail.smtp.starttls.enable", "true");
Session session = Session.getInstance(prop,
new javax.mail.Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
return new PasswordAuthentication(username, password);
}
});
try {
Message message = new MimeMessage(session);
message.setFrom(new InternetAddress("username#gmail.com"));
message.setRecipients(
Message.RecipientType.TO,
InternetAddress.parse("receiver address"));
message.setSubject("This mail is a test mail");
BodyPart messageBodyPart = new MimeBodyPart();
messageBodyPart.setText("Message");
Multipart multipart = new MimeMultipart();
multipart.addBodyPart(messageBodyPart);
messageBodyPart = new MimeBodyPart();
String filename = <Path>;
DataSource source = new FileDataSource(filename);
messageBodyPart.setDataHandler(new DataHandler(source));
messageBodyPart.setFileName(filename);
multipart.addBodyPart(messageBodyPart);
message.setContent(multipart);
Transport.send(message);
} catch (MessagingException e) {
JOptionPane.showMessageDialog(null, e.getMessage());
}
Is there any way of sending files greater than gmail's limit of 25 MB?
Can I change the file size upload limit from account settings or can I do something like any file will be uploaded as a drive link?
The user friendly way is probably to upload the file somewhere and add a link to the file in the mail.
It's also possible to split the file in several smaller parts and send each part in its own mail. The recipient then needs to join the files together again.
Zip-archivers can usually split large files into several zip-files that can then be join together again.
There's also raw splitting and joining. I haven't come across split commands built in to operating system distributions as standard. But your program could split the file in any way you desire.
Joining the files is then quite easy under Windows or Unix (Linux and others) operating systems. In Windows you go to the command prompt and use "copy": copy file1+file2+file3 finalfile In Unix you use "cat": cat file1 file2 file3 > finalfile
No. That is hard limit. Gmail itself states:
If your file is greater than 25 MB, Gmail automatically adds a Google Drive link in the email instead of including it as an attachment
That is what I would recommend for you as well, upload the file somewhere and paste a link in the mail. Options may be: Google Drive, Mega, Dropbox, S3, ...
Other than that there is nothing you can do.

How to find the Flat file which is currently updating record into it

In SSIS
In a folder there are many flat files and by using for each loop container we are processing it one by one. If any new file is placed in the folder and it is still in copying mode. Then, We should not take it for continue process. We should process Only fully copied file alone to our next process.
How can we achieve this? Please give your suggestions.
Best way I have done this in the past is to use a C# Script Task and try to open the file - If the file is still being copied you will get an error (which you Catch). Then you can set a boolean variable to conditionally process the file if the Open worked.
EG:
Boolean b = true;
FileStream f;
try
{
f = new FileStream("C:\\Test\\Test.txt", FileMode.Open, FileAccess.ReadWrite, FileShare.None);
}
catch (IOException e)
{
if (e.Message == "hello")
{
b = false;
}
}

Windows Phone 8 choose text file C#

i have a question. If there is a possibility at windows phone 8 at visual studio to create button event to read text file? i know about streamReader and if i declare wchich exacly file i want to read, but if i want to choose from list of files wchich i want to display. i did research on the Internet but i didint find an answer. I know i can use isolatedStorage to read music, video, image but not text files, on the app i created few files with text in it and i want users to have posibility to display one from this file, whichever they want to see. So, can you tell me how to do this?
You can use IsolatedStorage to read any file type you wish. You must of been using something like a Launcher that filters out the file type based on the Chooser.
You can open a file like this:
private async Task<string> ReadTextFile(string file_name)
{
// return buffer
string file_content = "";
// Get the local folder
StorageFolder local = Windows.Storage.ApplicationData.Current.LocalFolder;
if (local != null)
{
// Get the file
StorageFile file;
try
{
file = await local.GetFileAsync(file_name);
}
catch (Exception ex)
{
// no file, return empty
return file_content;
}
// Get the stream
System.IO.Stream file_stream = await file.OpenStreamForReadAsync();
// Read the data
using (StreamReader streamReader = new StreamReader(file_stream))
{
file_content = streamReader.ReadToEnd(); // read the full text file
streamReader.Close();
}
// Close the stream
file_stream.Close();
}
// return
return file_content;
}
If you want to get the PackageLocation (files that you added into the project like assets and images) then replace the LocalFolder with
Windows.ApplicationModel.Package package = Windows.ApplicationModel.Package.Current;
Windows.Storage.StorageFolder installedLocation = package.InstalledLocation;
With Windows Phone 8.1, File Pickers are allowed, consisting the same functionality you are expecting, so probably you might want to upgrade your app to WP8.1.
Here's more info on this API : Working with File Pickers

How do I SSIS WMI Event Watcher Query for a network folder?

What I'm trying to do in SSIS is have a WMI Event Watcher Task which watches a folder for a file to be created, then does something with it. The primary part is the "watching the folder for file creation".
I have a network folder (full path): \\srvblah10\main\child\target\
All the sites I've gone to has this as an example:
SELECT * FROM __InstanceCreationEvent WITHIN 10
WHERE TargetInstance ISA "CIM_DirectoryContainsFile"
AND TargetInstance.GroupComponent = "Win32_Directory.Name=\"d:\\\\NewFiles\""
Since the folder is a network folder, I can't provide the physical disk letter. So is there a way to use a similar WQL query but for network folder paths as opposed to physical folder paths?
You have to map the drive with a dos command:
net use s: \srvblah10\main\child\target\ /user dotnetN00b Pa$$word
then you can the WMI Event Watcher Task to watch it.
I was trying to do this for awhile, and finally gave up on trying to use the SSIS WMI Event Watcher task, and just wrote the equivalent in a Script task. The issue that was the challenge was getting the WMI Event Watcher to make the remote connection with specific user credentials that I wanted to obtain from a configuration section (not hard code into the package).
The second issue that was going to make not using a script difficult was simply translating the network share, into the local path name on the server, which the Event Watcher requires. You'll see from the scrip below, everything is accomplished with a minimal of effort.
Just an additional heads up, make sure to include the DTS.Variables the script uses in the ReadOnlyVariables (as normal). The code below requires three DTS variables, for example if you are trying to watch for files being dropped in the following location \copernicus\dropoff\SAP\Import, then you would set the variables as shown below:
User::ServerName - the hostname of the server where the share lives
(copernicus)
User::ShareName - the name of the network share
(dropoff)
User::ImportPath - the directory path of the directory to
watch for new files in (/SAP/Import)
public void Main()
{
string localPath = "";
try
{
ConnectionOptions connection = new ConnectionOptions();
connection.Username = "<valid username here>";
connection.Password = "<password here>";
connection.Authority = "ntlmdomain:<your domain name here>";
ManagementScope scope = new ManagementScope(#"\\" + Dts.Variables["User::FileServerName"].Value.ToString() + #"\root\CIMV2", connection);
scope.Connect();
/// Retrieve the local path of the network share from the file server
///
string queryStr = string.Format("SELECT Path FROM Win32_Share WHERE Name='{0}'", Dts.Variables["User::ShareName"].Value.ToString());
ManagementObjectSearcher mosLocalPath = new ManagementObjectSearcher(scope, new ObjectQuery(queryStr));
foreach (ManagementObject elements in mosLocalPath.Get())
{
localPath = elements["Path"].ToString();
}
queryStr = string.Format(
"SELECT * FROM __InstanceCreationEvent WITHIN 10 WHERE Targetinstance ISA 'CIM_DirectoryContainsFile' and TargetInstance.GroupComponent=\"Win32_Directory.Name='{0}{1}'\"",
localPath.Replace(#"\", #"\\"),
Dts.Variables["User::ImportPath"].Value.ToString().Replace(#"\", #"\\")); // query requires each seperator to be a double back slash
ManagementEventWatcher watcher = new ManagementEventWatcher(scope, new WqlEventQuery(queryStr));
ManagementBaseObject eventObj = watcher.WaitForNextEvent();
// Cancel the event subscription
watcher.Stop();
Dts.TaskResult = (int)ScriptResults.Success;
}
catch (ManagementException err)
{
Dts.Events.FireError((int)err.ErrorCode, "WMI File Watcher", "An error occurred while trying to receive an event: " + err.Message, String.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
catch (System.UnauthorizedAccessException unauthorizedErr)
{
Dts.Events.FireError((int)ManagementStatus.AccessDenied, "WMI File Watcher", "Connection error (user name or password might be incorrect): " + unauthorizedErr.Message, String.Empty, 0);
Dts.TaskResult = (int)ScriptResults.Failure;
}
}

Calling wkhtmltopdf to generate PDF from HTML

I'm attempting to create a PDF file from an HTML file. After looking around a little I've found: wkhtmltopdf to be perfect. I need to call this .exe from the ASP.NET server. I've attempted:
Process p = new Process();
p.StartInfo.UseShellExecute = false;
p.StartInfo.FileName = HttpContext.Current.Server.MapPath("wkhtmltopdf.exe");
p.StartInfo.Arguments = "TestPDF.htm TestPDF.pdf";
p.Start();
p.WaitForExit();
With no success of any files being created on the server. Can anyone give me a pointer in the right direction? I put the wkhtmltopdf.exe file at the top level directory of the site. Is there anywhere else it should be held?
Edit: If anyone has better solutions to dynamically create pdf files from html, please let me know.
Update:
My answer below, creates the pdf file on the disk. I then streamed that file to the users browser as a download. Consider using something like Hath's answer below to get wkhtml2pdf to output to a stream instead and then send that directly to the user - that will bypass lots of issues with file permissions etc.
My original answer:
Make sure you've specified an output path for the PDF that is writeable by the ASP.NET process of IIS running on your server (usually NETWORK_SERVICE I think).
Mine looks like this (and it works):
/// <summary>
/// Convert Html page at a given URL to a PDF file using open-source tool wkhtml2pdf
/// </summary>
/// <param name="Url"></param>
/// <param name="outputFilename"></param>
/// <returns></returns>
public static bool HtmlToPdf(string Url, string outputFilename)
{
// assemble destination PDF file name
string filename = ConfigurationManager.AppSettings["ExportFilePath"] + "\\" + outputFilename + ".pdf";
// get proj no for header
Project project = new Project(int.Parse(outputFilename));
var p = new System.Diagnostics.Process();
p.StartInfo.FileName = ConfigurationManager.AppSettings["HtmlToPdfExePath"];
string switches = "--print-media-type ";
switches += "--margin-top 4mm --margin-bottom 4mm --margin-right 0mm --margin-left 0mm ";
switches += "--page-size A4 ";
switches += "--no-background ";
switches += "--redirect-delay 100";
p.StartInfo.Arguments = switches + " " + Url + " " + filename;
p.StartInfo.UseShellExecute = false; // needs to be false in order to redirect output
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardError = true;
p.StartInfo.RedirectStandardInput = true; // redirect all 3, as it should be all 3 or none
p.StartInfo.WorkingDirectory = StripFilenameFromFullPath(p.StartInfo.FileName);
p.Start();
// read the output here...
string output = p.StandardOutput.ReadToEnd();
// ...then wait n milliseconds for exit (as after exit, it can't read the output)
p.WaitForExit(60000);
// read the exit code, close process
int returnCode = p.ExitCode;
p.Close();
// if 0 or 2, it worked (not sure about other values, I want a better way to confirm this)
return (returnCode == 0 || returnCode == 2);
}
I had the same problem when i tried using msmq with a windows service but it was very slow for some reason. (the process part).
This is what finally worked:
private void DoDownload()
{
var url = Request.Url.GetLeftPart(UriPartial.Authority) + "/CPCDownload.aspx?IsPDF=False?UserID=" + this.CurrentUser.UserID.ToString();
var file = WKHtmlToPdf(url);
if (file != null)
{
Response.ContentType = "Application/pdf";
Response.BinaryWrite(file);
Response.End();
}
}
public byte[] WKHtmlToPdf(string url)
{
var fileName = " - ";
var wkhtmlDir = "C:\\Program Files\\wkhtmltopdf\\";
var wkhtml = "C:\\Program Files\\wkhtmltopdf\\wkhtmltopdf.exe";
var p = new Process();
p.StartInfo.CreateNoWindow = true;
p.StartInfo.RedirectStandardOutput = true;
p.StartInfo.RedirectStandardError = true;
p.StartInfo.RedirectStandardInput = true;
p.StartInfo.UseShellExecute = false;
p.StartInfo.FileName = wkhtml;
p.StartInfo.WorkingDirectory = wkhtmlDir;
string switches = "";
switches += "--print-media-type ";
switches += "--margin-top 10mm --margin-bottom 10mm --margin-right 10mm --margin-left 10mm ";
switches += "--page-size Letter ";
p.StartInfo.Arguments = switches + " " + url + " " + fileName;
p.Start();
//read output
byte[] buffer = new byte[32768];
byte[] file;
using(var ms = new MemoryStream())
{
while(true)
{
int read = p.StandardOutput.BaseStream.Read(buffer, 0,buffer.Length);
if(read <=0)
{
break;
}
ms.Write(buffer, 0, read);
}
file = ms.ToArray();
}
// wait or exit
p.WaitForExit(60000);
// read the exit code, close process
int returnCode = p.ExitCode;
p.Close();
return returnCode == 0 ? file : null;
}
Thanks Graham Ambrose and everyone else.
OK, so this is an old question, but an excellent one. And since I did not find a good answer, I made my own :) Also, I've posted this super simple project to GitHub.
Here is some sample code:
var pdfData = HtmlToXConverter.ConvertToPdf("<h1>SOO COOL!</h1>");
Here are some key points:
No P/Invoke
No creating of a new process
No file-system (all in RAM)
Native .NET DLL with intellisense, etc.
Ability to generate a PDF or PNG (HtmlToXConverter.ConvertToPng)
Check out the C# wrapper library (using P/Invoke) for the wkhtmltopdf library: https://github.com/pruiz/WkHtmlToXSharp
There are many reason why this is generally a bad idea. How are you going to control the executables that get spawned off but end up living on in memory if there is a crash? What about denial-of-service attacks, or if something malicious gets into TestPDF.htm?
My understanding is that the ASP.NET user account will not have the rights to logon locally. It also needs to have the correct file permissions to access the executable and to write to the file system. You need to edit the local security policy and let the ASP.NET user account (maybe ASPNET) logon locally (it may be in the deny list by default). Then you need to edit the permissions on the NTFS filesystem for the other files. If you are in a shared hosting environment it may be impossible to apply the configuration you need.
The best way to use an external executable like this is to queue jobs from the ASP.NET code and have some sort of service monitor the queue. If you do this you will protect yourself from all sorts of bad things happening. The maintenance issues with changing the user account are not worth the effort in my opinion, and whilst setting up a service or scheduled job is a pain, its just a better design. The ASP.NET page should poll a result queue for the output and you can present the user with a wait page. This is acceptable in most cases.
You can tell wkhtmltopdf to send it's output to sout by specifying "-" as the output file.
You can then read the output from the process into the response stream and avoid the permissions issues with writing to the file system.
My take on this with 2018 stuff.
I am using async. I am streaming to and from wkhtmltopdf. I created a new StreamWriter because wkhtmltopdf is expecting utf-8 by default but it is set to something else when the process starts.
I didn't include a lot of arguments since those varies from user to user. You can add what you need using additionalArgs.
I removed p.WaitForExit(...) since I wasn't handling if it fails and it would hang anyway on await tStandardOutput. If timeout is needed, then you would have to call Wait(...) on the different tasks with a cancellationtoken or timeout and handle accordingly.
public async Task<byte[]> GeneratePdf(string html, string additionalArgs)
{
ProcessStartInfo psi = new ProcessStartInfo
{
FileName = #"C:\Program Files\wkhtmltopdf\wkhtmltopdf.exe",
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardInput = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
Arguments = "-q -n " + additionalArgs + " - -";
};
using (var p = Process.Start(psi))
using (var pdfSream = new MemoryStream())
using (var utf8Writer = new StreamWriter(p.StandardInput.BaseStream,
Encoding.UTF8))
{
await utf8Writer.WriteAsync(html);
utf8Writer.Close();
var tStdOut = p.StandardOutput.BaseStream.CopyToAsync(pdfSream);
var tStdError = p.StandardError.ReadToEndAsync();
await tStandardOutput;
string errors = await tStandardError;
if (!string.IsNullOrEmpty(errors)) { /* deal/log with errors */ }
return pdfSream.ToArray();
}
}
Things I haven't included in there but could be useful if you have images, css or other stuff that wkhtmltopdf will have to load when rendering the html page:
you can pass the authentication cookie using --cookie
in the header of the html page, you can set the base tag with href pointing to the server and wkhtmltopdf will use that if need be
Thanks for the question / answer / all the comments above. I came upon this when I was writing my own C# wrapper for WKHTMLtoPDF and it answered a couple of the problems I had. I ended up writing about this in a blog post - which also contains my wrapper (you'll no doubt see the "inspiration" from the entries above seeping into my code...)
Making PDFs from HTML in C# using WKHTMLtoPDF
Thanks again guys!
The ASP .Net process probably doesn't have write access to the directory.
Try telling it to write to %TEMP%, and see if it works.
Also, make your ASP .Net page echo the process's stdout and stderr, and check for error messages.
Generally return code =0 is coming if the pdf file is created properly and correctly.If it's not created then the value is in -ve range.
using System;
using System.Diagnostics;
using System.Web;
public partial class pdftest : System.Web.UI.Page
{
protected void Page_Load(object sender, EventArgs e)
{
}
private void fn_test()
{
try
{
string url = HttpContext.Current.Request.Url.AbsoluteUri;
Response.Write(url);
ProcessStartInfo startInfo = new ProcessStartInfo();
startInfo.FileName =
#"C:\PROGRA~1\WKHTML~1\wkhtmltopdf.exe";//"wkhtmltopdf.exe";
startInfo.Arguments = url + #" C:\test"
+ Guid.NewGuid().ToString() + ".pdf";
Process.Start(startInfo);
}
catch (Exception ex)
{
string xx = ex.Message.ToString();
Response.Write("<br>" + xx);
}
}
protected void btn_test_Click(object sender, EventArgs e)
{
fn_test();
}
}