Here headers are also inserting into database .here uploading the csv file with comma separated data
string Feedback = string.Empty;
string connString = ConfigurationManager.ConnectionStrings["DataBaseConnectionString"].ConnectionString;
using (MySqlConnection conn = new MySqlConnection(connString))
{
var copy = new MySqlBulkLoader(conn);
conn.Open();
try
{
copy.TableName = "BulkImportDetails";
copy.FileName = fileName;
copy.FieldTerminator = ",";
copy.LineTerminator = #"\n";
copy.Load();
Feedback = "Upload complete";
}
catch (Exception ex)
{
Feedback = ex.Message;
}
finally { conn.Close(); }
}
return Feedback;
Use the NumberOfLinesToSkip property to skip the first line, like so:
copy.NumberOfLinesToSkip = 1;
The use of this property is clearly shown in the documentation for MySQLBulkLoader. You must make a habit of reading the documentation to resolve your queries before you post a question here.
Related
I am working on c# utility to migrate data from SQL server 2017 to MongoDB. Below are steps I am following
1) Getting data from SQL server in JSON format (FOR JSON AUTO)
2) Parsing into BSON document
3) Then trying to insert into MongoDB
But I am getting error while reading JSON data from SQL.
My Json data is combination of root attributes as well as nested objects.
So Its dynamic data, that I want to PUSH as it is to MongoDB.
string jsonData = string.Empty;
foreach (var userId in userIdList)
{
using (SqlConnection con = new SqlConnection("Data Source=;Initial Catalog=;Integrated Security=True"))
{
using (SqlCommand cmd = new SqlCommand("Usp_GetUserdata", con))
{
cmd.CommandType = CommandType.StoredProcedure;
cmd.Parameters.Add("#userId", SqlDbType.Int).Value = userId;
con.Open();
var reader = cmd.ExecuteReader();
jsonResult = new StringBuilder();
//cmd.ExecuteNonQuery();
if (!reader.HasRows)
{
jsonResult.Append("[]");
}
else
{
while (reader.Read())
{
jsonResult.Append(reader.GetValue(0));
jsonData = reader.GetValue(0).ToString();
File.WriteAllText(#"c:\a.txt", jsonResult.ToString());
File.WriteAllText(#"c:\a.txt",jsonData);
jsonData.TrimEnd(']');
jsonData.TrimStart('[');
//Create client connection to our MongoDB database
var client = new MongoClient(MongoDBConnectionString);
//Create a session object that is used when leveraging transactions
var session = client.StartSession();
//Create the collection object that represents the "products" collection
var employeeCollection = session.Client.GetDatabase("mongodev").GetCollection<BsonDocument>("EmpData");
//Begin transaction
session.StartTransaction();
try
{
dynamic resultJson = JsonConvert.DeserializeObject(result);
var document = BsonSerializer.Deserialize<BsonDocument>(resultJson);
//MongoDB.Bson.BsonDocument document
// = MongoDB.Bson.Serialization.BsonSerializer.Deserialize<BsonDocument>(jsonResult);
employeeCollection.InsertOneAsync(document);
//BsonArray pipeline =
// MongoDB.Bson.Serialization.BsonSerializer.Deserialize<BsonArray>(jsonData);
//var documents = pipeline.Select(val => val.AsBsonDocument);
//employeeCollection.InsertManyAsync(documents);
session.CommitTransaction();
}
catch (Exception e)
{
Console.WriteLine(e);
session.AbortTransaction();
throw;
}
}
}
}
}
}
I want to send an email with csv attachment without having to store the csv file on the server.
I create my csv :
StringWriter sw = new StringWriter();
try {
CSVPrinter csvPrinter = new CSVPrinter(sw, CSVFormat.DEFAULT.withHeader(
"Année", "Mois", "Date", "Pièce", "Libellé",
"Débit", "Crédit", "Compte", "Journal"
));
for (ExportComptaAV export : exports){
csvPrinter.printRecord(export.getAnnee().getLibelle(),
export.getMois().getLibelle(),
export.getDateMouvement().format(DateTimeFormatter.ofPattern("yyyy-MM-dd")),
export.getPiece(),
export.getLibelle(),
export.getMontantDebit().toString(),
export.getMontantCredit().toString(),
export.getNumCompte(),
export.getCodeJournal());
}
} catch (IOException e) {
e.printStackTrace();
}
And my email procedure :
MimeMessage mimeMessage = javaMailSender.createMimeMessage();
try {
MimeMessageHelper message = new MimeMessageHelper(mimeMessage, true, CharEncoding.UTF_8);
InternetAddress ad = new InternetAddress("test#gmail.com");
message.addTo(ad);
message.setSubject(sujet);
message.setText(content, isHtml);
String fileName = "csvFile.csv";
message.addAttachment(fileName, new ByteArrayResource(...));
javaMailSender.send(mimeMessage);
} catch (MessagingException mex) {
throw mex;
}
So I would like to convert my CsvPrinter to a ByteArrayresource.
Is there a way to do that?
Get the String from the StringWriter, get the bytes from the String, and use them with ByteArrayResource.
Please keep in mind this is a open question and I am not looking for a specific answer but just approaches and routes I can take.
Essentially I am getting a csv file from my aws s3 bucket. I am able to get it successfully using
AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
S3Object object = s3Client.getObject(
new GetObjectRequest(bucketName, key));
Now I want to populate a dynamodb table using this JSON file.
I was confused as i found all sorts of stuff online.
Here is one suggestion - This approach is however only reading the file it is not inserting anything to the dynamodb table.
Here is another suggestion - This approach is lot closer to what i am looking for , it is populating a table from a JSON file.
However i was wondering is there a generic way to ready any json file and populate a dynamodb table based on that ? Also for my case what approach is the best?
Since i originally asked the question I did more work.
What I have done so far
I have a csv file sitting in s3 that looks like this
name,position,points,assists,rebounds
Lebron James,SF,41,12,11
Kyrie Irving,PG,41,7,5
Stephen Curry,PG,29,8,4
Klay Thompson,SG,31,5,5
I am able to sucessfully pick it up as a s3object doing the following
AmazonS3 s3client = new AmazonS3Client(/**new ProfileCredentialsProvider()*/);
S3Object object = s3client.getObject(
new GetObjectRequest("lambda-function-bucket-blah-blah", "nba.json"));
InputStream objectData = object.getObjectContent();
Now I want to insert this in to my dynamodb table so i am attempting the following.
AmazonDynamoDBClient dbClient = new AmazonDynamoDBClient();
dbClient.setRegion(Region.getRegion(Regions.US_BLAH_1));
DynamoDB dynamoDB = new DynamoDB(dbClient);
//DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("MyTable");
//after this point i have tried many json parsers etc and did table.put(item) etc but nothing has worked. I would appreciate kind help
For CSV parsing, you can use plain reader as your file looks quite simple
AmazonS3 s3client = new AmazonS3Client(/**new ProfileCredentialsProvider()*/);
S3Object object = s3client.getObject(
new GetObjectRequest("lambda-function-bucket-blah-blah", "nba.json"));
InputStream objectData = object.getObjectContent();
AmazonDynamoDBClient dbClient = new AmazonDynamoDBClient();
dbClient.setRegion(Region.getRegion(Regions.US_BLAH_1));
DynamoDB dynamoDB = new DynamoDB(dbClient);
//DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("MyTable");
String line = "";
String cvsSplitBy = ",";
try (BufferedReader br = new BufferedReader(
new InputStreamReader(objectData, "UTF-8"));
while ((line = br.readLine()) != null) {
// use comma as separator
String[] elements = line.split(cvsSplitBy);
try {
table.putItem(new Item()
.withPrimaryKey("name", elements[0])
.withString("position", elements[1])
.withInt("points", elements[2])
.....);
System.out.println("PutItem succeeded: " + elements[0]);
} catch (Exception e) {
System.err.println("Unable to add user: " + elements);
System.err.println(e.getMessage());
break;
}
}
} catch (IOException e) {
e.printStackTrace();
}
Depending the complexity of your CSV, you can use 3rd party libraries like Apache CSV Parser or open CSV
I leave the original answer for parsing JSon
I would use the Jackson library and following your code do the following
AmazonS3 s3client = new AmazonS3Client(/**new ProfileCredentialsProvider()*/);
S3Object object = s3client.getObject(
new GetObjectRequest("lambda-function-bucket-blah-blah", "nba.json"));
InputStream objectData = object.getObjectContent();
AmazonDynamoDBClient dbClient = new AmazonDynamoDBClient();
dbClient.setRegion(Region.getRegion(Regions.US_BLAH_1));
DynamoDB dynamoDB = new DynamoDB(dbClient);
//DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("MyTable");
JsonParser parser = new JsonFactory()
.createParser(objectData);
JsonNode rootNode = new ObjectMapper().readTree(parser);
Iterator<JsonNode> iter = rootNode.iterator();
ObjectNode currentNode;
while (iter.hasNext()) {
currentNode = (ObjectNode) iter.next();
String lastName = currentNode.path("lastName").asText();
String firstName = currentNode.path("firstName").asText();
int minutes = currentNode.path("minutes").asInt();
// read all attributes from your JSon file
try {
table.putItem(new Item()
.withPrimaryKey("lastName", lastName, "firstName", firstName)
.withInt("minutes", minutes));
System.out.println("PutItem succeeded: " + lastName + " " + firstName);
} catch (Exception e) {
System.err.println("Unable to add user: " + lastName + " " + firstName);
System.err.println(e.getMessage());
break;
}
}
parser.close();
Inserting the records in your table will depend of your schema, I just put an arbitrary example, but anyway this will get you the reading of your file and the way to insert into the dynamoDB table
As you talked about the different approaches, another possibility is to setup a AWS Pipeline
I want to read file using StreamResourceInfo GetResourceStream(Uri uriResource) method where my filename is Assets and its type if file (extension) so I used following line of code in windows phone 8.1 sdk,
StreamResourceInfo info = App.GetResourceStream(new Uri("Assets", UriKind.Relative));
But the info variable shows null value.
Is it necessary to use GetResourceStream in your case? Your question is a little bit unclear, but if you want to get the content of the file you can try this:
public async Task<string> GetFileContent(string fileName)
{
try
{
string text = string.Empty;
StorageFile storageFile = await Windows.ApplicationModel.Package.Current.InstalledLocation.GetFileAsync(fileName);
if (storageFile != null)
{
IBuffer buffer = await FileIO.ReadBufferAsync(storageFile);
DataReader reader = DataReader.FromBuffer(buffer);
byte[] fileContent = new byte[reader.UnconsumedBufferLength];
reader.ReadBytes(fileContent);
text = Encoding.UTF8.GetString(fileContent, 0, fileContent.Length);
}
return text;
}
catch (Exception e)
{
return string.Empty;
}
}
Note, it returns string UTF8 encoded, you can change it if necessary.
usage:
var fileContent = await GetFileContent(#"Assets\[yourfile]");
I am reading data from IsolatedStorage, but can't edit it in ScheduledTask. How can I edit it?
private void StartToastTask(ScheduledTask task)
{
long rank = 0, difference = 0;
string text = "", nickname = "";
PishtiWCF.PishtiWCFServiceClient ws = ServiceClass.GetPishtiWCFSvc();
ws.GetUsersRankCompleted += (src, e) =>
{
try
{
if (e.Error == null)
{
difference = rank - e.Result.GeneralRank;
if (!String.IsNullOrEmpty(nickname))
{
if (difference < 0)
text = string.Format("{0}, {1} kişi seni geçti!", nickname, difference.ToString(), e.Result.GeneralRank);
else if (difference > 0)
text = string.Format("{0}, {1} kişiyi daha geçtin!", nickname, Math.Abs(difference).ToString(), e.Result.GeneralRank);
else if (e.Result.GeneralRank != 1)
text = string.Format("{0}, sıralamadaki yerin değişmedi!", nickname, e.Result.GeneralRank);
else
text = string.Format("{0}, en büyük sensin, böyle devam!", nickname);
}
else
return;
Mutex mut;
if (!Mutex.TryOpenExisting("IsoStorageMutex", out mut))
mut = new Mutex(false, "IsoStorageMutex");
mut.WaitOne();
using (IsolatedStorageFile file = IsolatedStorageFile.GetUserStoreForApplication())
{
using (IsolatedStorageFileStream stream = file.OpenFile("UserRanks", FileMode.Open, FileAccess.Write))
{
StreamWriter writer = new StreamWriter(stream);
writer.Write(string.Format("{0},{1}", nickname, e.Result.GeneralRank));
writer.Close();
stream.Close();
}
}
mut.ReleaseMutex();
ShellToast toast = new ShellToast();
toast.Title = "Pishti";
toast.Content = text;
toast.Show();
}
FinishTask(task);
}
catch (Exception)
{
}
};
try
{
Mutex mut;
if (!Mutex.TryOpenExisting("IsoStorageMutex", out mut))
mut = new Mutex(false, "IsoStorageMutex");
mut.WaitOne();
using (IsolatedStorageFile file = IsolatedStorageFile.GetUserStoreForApplication())
{
using (IsolatedStorageFileStream stream = file.OpenFile("UserRanks", FileMode.Open, FileAccess.Read))
{
using (StreamReader reader = new StreamReader(stream))
{
string temp = reader.ReadToEnd();
if (temp.Split(',').Count() > 1)
{
nickname = temp.Split(',')[0];
rank = long.Parse(temp.Split(',')[1]);
ws.GetUsersRankAsync(nickname);
}
reader.Close();
}
stream.Close();
}
}
mut.ReleaseMutex();
}
catch (Exception)
{
}
}
I am getting rank from UserRanks file, for example 1200, but when I get and data from WCF, edit it to 1000 and want to write it to IsolatedStorage, It doesn't crash application but it fails.
Do you know why?
Thanks.
I've fixed it with delete file.
Mutex mut;
if (!Mutex.TryOpenExisting("IsoStorageMutex", out mut))
mut = new Mutex(false, "IsoStorageMutex");
mut.WaitOne();
using (IsolatedStorageFile file = IsolatedStorageFile.GetUserStoreForApplication())
{
if (file.FileExists("UserRanks"))
file.DeleteFile("UserRanks");
using (IsolatedStorageFileStream stream = file.OpenFile("UserRanks", FileMode.OpenOrCreate, FileAccess.Write))
{
StreamWriter writer = new StreamWriter(stream);
writer.Write(string.Format("{0},{1}", nickname, e.Result.GeneralRank));
writer.Close();
stream.Close();
}
}
mut.ReleaseMutex();
You appear to write to the file first, which makes sense, but when you do so you use a file access mode - FileMode.Open - which means "open an existing file". The first time you do this the file won't exist and the open will fail.
You should either use FileMode.OpenOrCreate, which is self explanatory, or FileMode.Append which will open the file if it exists and seek to the end of the file, or create a new file if it doesn't.
If you want to throw away any pre-existing file (which is what your delete then create will do) then just use FileMode.Create