I have a huge collection of visual foxpro dbf files that I would like to convert to csv.
(If you like, you can download some of the data here. Click on the 2011 link for Transaction Data, and prepare to wait a long time...)
I can open each table with DBF View Plus (an awesome freeware utility), but exporting them to csv takes a few hours per file, and I have several dozen files to work with.
Is there a program like DBF View plus that will allow me to set up a batch of dbf-to-csv conversions to run overnight?
/Edit: Alternatively, is there a good way to import .dbf files straight into SQL Server 2008? They should all go into 1 table, as each file is just a subset of records from the same table and should have all the same column names.
Load up your list of FoxPro files in an array/list then call the ConvertDbf on each to convert them from FoxPro to csv files. See the c# console application code below...
Credit c# datatable to csv for the DataTableToCSV function.
using System;
using System.Data;
using System.Data.OleDb;
using System.IO;
using System.Linq;
using System.Text;
namespace SO8843066
{
class Program
{
static void Main(string[] args)
{
string connectionString = #"Provider=VFPOLEDB.1;Data Source=C:\";
string dbfToConvert = #"C:\yourdbffile.dbf";
ConvertDbf(connectionString, dbfToConvert, dbfToConvert.Replace(".dbf", ".csv"));
Console.WriteLine("End of program execution");
Console.WriteLine("Press any key to end");
Console.ReadKey();
}
static void DataTableToCSV(DataTable dt, string csvFile)
{
StringBuilder sb = new StringBuilder();
var columnNames = dt.Columns.Cast<DataColumn>().Select(column => column.ColumnName).ToArray();
sb.AppendLine(string.Join(",", columnNames));
foreach (DataRow row in dt.Rows)
{
var fields = row.ItemArray.Select(field => field.ToString()).ToArray();
for (int i =0;i < fields.Length;i++)
{
sb.Append("\"" + fields[i].Trim() );
sb.Append((i != fields.Length - 1) ? "\"," : "\"");
}
sb.Append("\r\n");
}
File.WriteAllText(csvFile, sb.ToString());
}
static void ConvertDbf(string connectionString, string dbfFile, string csvFile)
{
string sqlSelect = string.Format("SELECT * FROM {0}", dbfFile);
using (OleDbConnection connection = new OleDbConnection(connectionString))
{
using (OleDbDataAdapter da = new OleDbDataAdapter(sqlSelect, connection))
{
DataSet ds = new DataSet();
da.Fill(ds);
DataTableToCSV(ds.Tables[0], csvFile);
}
}
}
}
}
In that case, SQL-Server I think has a capability of connecting to foxpro tables. I'm not exactly sure how as I've never done it recently (last time using SQL-Server about 8+ yrs ago). I'm sure there are other threads out there that can point you to connecting SQL-Server to VFP.
I quickly searched and saw this thread
In addition, you might need the latest OleDb provider to establish the connection which I've also posted in a thread here. This thread also shows a sample of the connection string information you may need from SQL-Server. The data source information should point to the PATH where the .DBF files are found, and not the specific name of the .DBF you are trying to connect to.
Hope this helps you out.
This works very well and thanks for the solution. I used this to convert some visual foxpro dbf tables to flat files. With these tables, there is the additional challenge of converting fields of type Currency.
Currency fields are a 64-bit (8 byte) signed integer amidst a 36 element byte array starting at the 27th position. The integer is then divided by 1000 to get 4-decimal precision equivalent.
If you have this type of field, try this inside the fields FOR loop
if (("" + fields[i]).Equals("System.Byte[]"))
{
StringBuilder db = new StringBuilder();
byte[] inbytes = new byte[36];
inbytes = ObjectToByteArray(fields[i]);
db.Append("" + (double)BitConverter.ToInt64(inbytes,27)/1E4);
sb.Append("\"" + db);
}
With the following helper method
private static byte[] ObjectToByteArray(Object obj)
{
BinaryFormatter bf = new BinaryFormatter();
using (var ms = new MemoryStream())
{
bf.Serialize(ms, obj);
return ms.ToArray();
}
}
Check out my answer to Foxbase to postrgresql data transfer. (dbf files reader).
Related
Is there a way to reference a csv or other data source which has table names and source queries to generate BIML documents?
Thanks
Do be aware C# is not my strong suit and the below may well not be the ideal way to do this. If you do find something more suitable, I would very much like to hear about it :)
The easiest way I have found to include CSV based metadata into my Biml projects is to load them into C# DataTable objects that I then reference in my Biml as
a C# variable object, which plays very well with foreach to iterate through the rows.
Assuming you are aware how to include C# in your Biml projects (either in the file directly or via referenced .cs file), you can use the following code:
public static DataTable FlatFileToDataTable(string filePath, char delimiter)
{
DataTable dt = new DataTable();
using (StreamReader sr = new StreamReader(filePath))
{
string[] headers = sr.ReadLine().Split(delimiter);
foreach (string header in headers)
{
dt.Columns.Add(header);
}
while (!sr.EndOfStream)
{
string[] rows = sr.ReadLine().Split(delimiter);
DataRow dr = dt.NewRow();
for (int i = 0; i < headers.Length; i++)
{
dr[i] = rows[i];
}
dt.Rows.Add(dr);
}
}
return dt;
}
I think in order to use the StreamReader you will need to add using System.IO; to your code file as well.
Usage would be to define a DataTable object and populate it with the result of the above, then to reference it using code snippets within your Biml:
DataTable YourDataTable = FlatFileToDataTable("<Path to CSV file>",'<Value Delimiter>');
...
<Columns>
<# foreach(DataRow r in YourDataTable.Rows){ #>
<Column Name="<#=r["YourColumnName"]#>" etc />
<# } #>
</Columns>
I am trying to convert a JSON file into CSV.
I have a JAVA code which is able to do it perfectly on UNIX file system and on local file system.
I have written below main class to perform this conversion on HDFS.
public class ClassMain {
public static void main(String[] args) throws IOException {
String uri = args[1];
String uri1 = args[2];
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri), conf);
FSDataInputStream in = null;
FSDataOutputStream out = fs.create(new Path(uri1));
try{
in = fs.open(new Path(uri));
JsonToCSV toCSV = new JsonToCSV(uri);
toCSV.json2Sheet().write2csv(uri1);
IOUtils.copyBytes(in, out, 4096, false);
}
finally{
IOUtils.closeStream(in);
IOUtils.closeStream(out);
}
}
}
json2sheet and write2csv are methods which perform the conversion and write operation.
I am running this jar using below command:
hadoop jar json-csv-hdfs.jar com.nishant.ClassMain /nishant/large.json /nishant/output
The problem is, it does not write anything at /nishant/output. It creates a 0 sized /nishant/output file.
Maybe the usage of copyBytes is not a good idea here.
How to achieve this on HDFS if it is working OK on unix FS and local FS.
Here I am trying to convert JSON file to CSV and not trying to map JSON objects to their values
FileSystem needs only one configuration key to successfully connect to HDFS.
conf.set(key, "hdfs://host:port"); // where key="fs.default.name"|"fs.defaultFS"
I use something like this for my application
MySqlConnection cnn = new MySqlConnection("Server=myServerAddress;" +
"Database=myDataBase;" +
"Uid=myUsername;" +
"Pwd=myPassword;");
And this changes everytime because we deploy databases with our application.
It works fine. I type in using(new connection(cnn)){ query... } and go.
And I've got it working with a dataset using a connection defined in the windows ODBC datasouce administrator.
But I'm curious, is there a way to use visual studio's dataset items using the my local test db and then change the connection of the dataset at runtime? Even better, can I use c# to programmatically add the ODBC data source at runtime?
Usually a connection string is loaded from the application exe.config file present in the same folder of the application. This connection string could be defined using the Settings tab in the project properties.
Right click on Properties of your project
Select the Settings tab (confirm the creation if you have no
settings)
Click on the ComboBox in the column type and select Connection String
Give a symbolic name to your connection
Type the connection string in the Value column (Examples at
connectionstrings.com)
Now in your project files you should have the file app.config (that becomes yourapp.exe.config) where there is a section like this
<configuration>
<connectionStrings>
<add name="MyAppConnection"
connectionString="Server=myServerAddress;Database=myDB;Uid=user;Pwd=pass;" />
</connectionStrings>
</configuration
At this point you read it in the program using
string conString = ConfigurationManager
.ConnectionStrings["MyAppConnection"]
.ConnectionString;
Instead in a dynamic situation where you want to build yourself the connection string during runtime (from user inputs, your own configuration files and so on) then you could leverage the functionality of the class MySqlConnectionStringBuilder
MySqlConnectionStringBuilder msb = new MySqlConnectionStringBuilder();
msb.Server = "localhost";
msb.Port = 3306;
msb.UserID = "root";
msb.Password = "xxx";
msb.Database = "test";
MySqlConnection cnn = new MySqlConnection(msb.ConnectionString);
cnn.Open();
Of course, these literal values could be substituted by your own variables.
The documentation of this class is surprising difficult to find. The best docs are the one of the Sql Server equivalent. It is interesting that you could read a static connection string from your config file and then change only the property needed.
string conString = ConfigurationManager
.ConnectionStrings["MyAppConnection"]
.ConnectionString;
MySqlConnectionStringBuilder msb = new MySqlConnectionStringBuilder(conString);
msb.Database = "AnotherDB";
MySqlConnection cnn = new MySqlConnection(msb.ConnectionString);
Application connection string cannot be changed at runtime.
User settings can be changed.
Assuming you are using an application setting-property named "MyConnectionString" which holds the connection string for the entire application.
On your main Program class create a global string:
internal static string Prconnstring;
Create and save this settings.cs file:
namespace MYSOLUTIONORPROJECTNAME.Properties
{
// (Not sure where I found this solution some time ago)
// This class allows you to handle specific events on the settings class:
// The SettingChanging event is raised before a setting's value is changed.
// The PropertyChanged event is raised after a setting's value is changed.
// The SettingsLoaded event is raised after the setting values are loaded.
// The SettingsSaving event is raised before the setting values are saved.
internal sealed partial class Settings
{
public Settings()
{
// // To add event handlers for saving and changing settings, uncomment the lines below:
//
// this.SettingChanging += this.SettingChangingEventHandler;
//
// this.SettingsSaving += this.SettingsSavingEventHandler;
//
}
private void SettingChangingEventHandler(object sender, System.Configuration.SettingChangingEventArgs e)
{
// Add code to handle the SettingChangingEvent event here.
}
private void SettingsSavingEventHandler(object sender, System.ComponentModel.CancelEventArgs e)
{
// Add code to handle the SettingsSaving event here.
}
public override object this[string propertyName]
{
get
{
if (propertyName == "MyConnectionString")
{
return Program.Prconnstring;
}
else
{
return base[propertyName];
}
}
set
{
base[propertyName] = value;
}
}
}
}
Before calling-opening any object that uses the connection string (examples include Forms that use datasets or other classes that use datasets created on the development enviroment) create your new connection string by any means you think. (Example: You might want to use as user name in the connection string the current user. Create the connection string using the info provided form the environment.)
Program.Prconnstring = thenewruntimeconnectionstring.
Now whenever the application tries to get MyConnectionString (which is hardcoded in the myapplicationname.config and cannot be changed) instead gets the new thenewruntimeconnectionstring you provided to Program.Prconnstring.
Be aware that the development connection string will be available-visible to final user, since it is just a text file. If you do not want this, you can change that file (will be a file named NAMEOFMYAPPLICATION.exe.config) during deployment, since the connection string hardcoded there, will be of no use for the running app. Do not delete it, just change.
Your connection string will be stored in your App.config (or c# equivalent). Say it's called MyConnectionString. Just add My.Settings("MyConnectionString")="[your new connection string]" to your entry point to change to database binding at runtime. E.g:
Public Sub New()
' This call is required by the designer.
InitializeComponent()
My.Settings("MyConnectionString") = "server=remotedb.uk;user id=MainUser;password=2jdi38edhnche73g;database=mainDb;persistsecurityinfo=True;allowuservariables=True;defaultcommandtimeout=480;characterset=utf8mb4"
End Sub
I want to create a JSON file for use as part of a simple web prototyping exercise. LinqPAD is perfect for accessing the data from my DB in just the shape I need, however I cannot get it out as JSON very easily.
I don't really care what the schema is, because I can adapt my JavaScript to work with whatever is returned.
Is this possible?
A more fluent solution is to add the following methods to the "My Extensions" File in Linqpad:
public static String DumpJson<T>(this T obj)
{
return
obj
.ToJson()
.Dump();
}
public static String ToJson<T>(this T obj)
{
return
new System.Web.Script.Serialization.JavaScriptSerializer()
.Serialize(obj);
}
Then you can use them like this in any query you like:
Enumerable.Range(1, 10)
.Select(i =>
new
{
Index = i,
IndexTimesTen = i * 10,
})
.DumpJson();
I added "ToJson" separately so it can be used in with "Expessions".
This is not directly supported, and I have opened a feature request here. Vote for it if you would also find this useful.
A workaround for now is to do the following:
Set the language to C# Statement(s)
Add an assembly reference (press F4) to System.Web.Extensions.dll
In the same dialog, add a namespace import to System.Web.Script.Serialization
Use code like the following to dump out your query as JSON
new JavaScriptSerializer().Serialize(query).Dump();
There's a solution with Json.NET since it does indented formatting, and renders Json dates properly. Add Json.NET from NuGet, and refer to Newtonsoft.Json.dll to your “My Extensions” query and as well the following code :
public static object DumpJson(this object value, string description = null)
{
return GetJson(value).Dump(description);
}
private static object GetJson(object value)
{
object dump = value;
var strValue = value as string;
if (strValue != null)
{
var obj = JsonConvert.DeserializeObject(strValue);
dump = JsonConvert.SerializeObject(obj, Newtonsoft.Json.Formatting.Indented);
}
else
{
dump = JsonConvert.SerializeObject(value, Newtonsoft.Json.Formatting.Indented);
}
return dump;
}
Use .DumpJson() as .Dump() to render the result. It's possible to override more .DumpJson() with different signatures if necessary.
As of version 4.47, LINQPad has the ability to export JSON built in. Combined with the new lprun.exe utility, it can also satisfy your needs.
http://www.linqpad.net/lprun.aspx
I have a question that I hope someone can answer. I am using .Net 3.5 Winforms and the SpreadSheetGear 2010 component and need to know if there is a free or low cost method to convert an Excel Workbook to HTML? Is there a good XSLT transform or some low cost or open source component? I can save the Excel file to OpenXMLWorkbook programmactically but the component does not allow to save to html. Any help would be appreciated. Thanks.
Use ASP.Net and follow the code demonstrated by the Excel to DataGrid Samples. You can always include a reference to System.Web in your project.
But, even without ASP.Net, and sticking with the .Net client profile, generating an HTML table from a DataTable is simple. Use the AntiXss Library (download) to encode the data to html. Then construct your HTML with a StringBuilder. Here's an extension method you can use to convert any DataTable to an html <table>:
using AntiXss = Microsoft.Security.Application;
...
public static String ToHTML(this DataTable dt)
{
StringBuilder html = new StringBuilder("<table><thead><tr>");
foreach (DataColumn col in dt.Columns)
{
html.AppendFormat("<td>{0}</td>",
AntiXss.Encoder.HtmlEncode(col.ColumnName));
}
html.Append("</tr></thead><tbody>");
foreach (DataRow row in dt.Rows)
{
html.Append("<tr>");
foreach (var data in row.ItemArray)
{
html.AppendFormat("<td>{0}</td>",
AntiXss.Encoder.HtmlEncode(data.ToString()));
}
html.Append("</tr>");
}
html.Append("</tbody></table>");
return html.ToString();
}
And to get html for an entire workbook, another extension method:
public static List<String> ToHTML(this IWorkBook wb)
{
List<String> tables = new List<String>();
DataSet ds = wb.GetDataSet(GetDataFlags.FormattedText);
foreach (DataTable dt in ds.Tables)
{
tables.Add(dt.ToHTML());
}
return tables;
}