I'm currently using couchbase-lite inside my iOS and android application to sync down files from a database running CouchDB.
Every so often I remove files that are not longer needed, and I would like the same files to be removed from the mobile app as well, but any pull replication only pulls updates or new files, and doesnt trigger a delete on the mobile app.
Is there any way to delete documents from the mobile app that are no longer on the server DB without doing a full purge on the mobile application, and then resyncing the whole database?
From the database class there is a purgeDocument() method. This removes the target document from the local database - server copies of target documents are left unchanged. If the document is later updated then the document will return to the local client on next replication.
http://docs.couchbase.com/mobile/2.1/couchbase-lite-swift/Classes/Database.html
You can delete documents from database based on id as following:
try {
for (Result result : docList) {
String id = result.getString(0);
Document doc = database.getDocument(id);
database.delete(doc);
}
} catch (CouchbaseLiteException e) {
e.printStackTrace();
}
To get the document id you have to query something like this:
Query query = QueryBuilder
.select(SelectResult.expression(Meta.id), SelectResult.all())
.from(DataSource.database(database))
ResultSet rs = query.execute();
for (Result result : rs) {
Dictionary data = result.getDictionary("db_name");
Log.e("CouchbaseLite ", "document: " + data);
Log.e("CouchbaseLite ", "id: " + result.getString(0));
}
Related
Using Entity Framework 6 and MySQL, I am trying to archive data from a 'production' database table to an 'archive' database. I have created two DBContexts one for each database. Each database has the same schema.
I can move an entire table of data from the production database to the archive database using the following code:
using (MyDBContext archiveContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("archive_db"))
using (MyDBContext prodContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("prod_db"))
{
if(prodContext.myTable.Any())
{
archiveContext.myTable.AddRange(prodContext.myTable.AsNoTracking());
archiveContext.SaveChanges();
}
}
However I don't want to archive the whole table, I only wish to archive data older than a certain date, so I tried the following:
using (MyDBContext archiveContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("archive_db"))
using (MyDBContext prodContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("prod_db"))
{
IQueryable<myTable> dataToArchive =
from mt in prodContext.myTable
where mt.date < DateTimeSixMonths
select mt;
archiveContext.myTable.AddRange(dataToArchive);
archiveContext.SaveChanges();
}
but I cannot get around the exception I get when I run this:
System.InvalidOperationException: 'An entity object cannot be
referenced by multiple instances of IEntityChangeTracker.'
It occurs on this line:
archiveContext.myTable.AddRange(dataToArchive);
Is it possible to somehow remove the tracking from the 'dataToArchive'
Have you tried disposing the first DataContext after retrieving data? Something like this:
List<myTable> dataToArchive;
using (MyDBContext prodContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("prod_db"))
{
dataToArchive = (from mt in prodContext.myTable
where mt.date < DateTimeSixMonths
select mt).ToList();
}
using (MyDBContext archiveContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("archive_db"))
{
archiveContext.myTable.AddRange(dataToArchive);
archiveContext.SaveChanges();
}
Using EF to manage archiving data isn't ideal, something like that would be better served at the database level using insert-select + delete for low to moderate data volumes or detachable partitions (I.e. 3-6 mo. partition sizes) that can be moved between databases.
To do this with EF (only recommended for small and non-complex domain models) you should be able to accomplish this by disabling the proxy generation in your read context, load the data AsNoTracking, then add it to the new context DbSet. This example does not handle associated entities, or do the delete from the prod DbSet.
using (MyDBContext prodContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("prod_db"))
{
prodContext.Configuration.ProxyCreationEnabled = false;
dataToArchive = prodContext.myTable.AsNoTracking()
.Where(mt => mt.Date < DateTimeSixMonths);
using (MyDBContext archiveContext =
MyDBContext.CreateEntitiesForSpecificDatabaseName("archive_db"))
{
archiveContext.myTable.AddRange(dataToArchive);
archiveContext.SaveChanges();
}
}
i have the following code which has been getting me data from flat files. but now all of a sudden i am getting this error
System.Data.OleDb.OleDbException: Invalid path or file name
but the code hasnt changed it worked for months,im not sure what went wrong.
System.Web.Script.Serialization.JavaScriptSerializer json = new System.Web.Script.Serialization.JavaScriptSerializer();
string jsonText;
System.Collections.Generic.List<object> objList = new List<object>();
string strConn = #"Provider=vfpoledb;Data Source=\\10.0.0.0\wwwroot\apps\assembly\FlatDatabaseDbfs\vt_Flat.dbf;Collating Sequence=machine;";
using (System.Data.OleDb.OleDbConnection conn = new System.Data.OleDb.OleDbConnection(strConn))
{
System.Data.OleDb.OleDbCommand cmddbf = new System.Data.OleDb.OleDbCommand();
cmddbf.Connection = conn;
conn.Open();
cmddbf.CommandText = "select * from vt_Flat";
var dr = cmddbf.ExecuteReader();
while (dr.Read())
{
objList.Add(new
{
Code = (dr["dp_code"].ToString().Trim()),
});
};
}
var filteredList = objList.Where(obj => ((dynamic)obj).Status == (Request.QueryString["Status"] ?? "") && ((dynamic)obj).DepCode == (Request.QueryString["Code"] ?? ""));
jsonText = json.Serialize(filteredList);
Response.Write(jsonText);
}
is there something wrong with iis permissions?
Aside from the connection having to point to the PATH as already noted by Oleg, in the C# instances of OleDbConnection I have done in the past, the connection string uses
Provider=VFPOLEDB.1
Don't know if it is case/sensitive issue and the ".1" which is also part of the provider string.
Once you have a valid connection to the PATH, then your query can query from any table within the path location. So if you had 2+ files, and needed to join them, you would do so with a standard query / join. In your case, your command text is only "select *" since you changed your original connection that included the table. Change the command text to
"select * from vt_Flat"
OTHER CONSIDERATIONS
Is this being run from some web service project? If so, THAT could be the basis. You as a developer testing are running with your permissions / access. If running as a web server, the WEB-based user account may not have permissions to the folder to process / work with the data.
Check the folder of your production data to ALLOW the web user if so running. If that doesn't work, set permissions on the folder to EVERYBODY (only for testing/confirmation purposes only). See if that is the problem.
Also, from the Provider connection, did you try with it as all upper case VFPOLEDB.1?
Use path instead of file name, e.g.:
Data Source=\\10.0.0.0\wwwroot\apps\assembly\FlatDatabaseDbfs\;
I'm working on a project where i need to do lookups on a data warehouse server in Integration Services.
My problem is that I need to be able to change what database it i performs the lookup to. The databases are design wise identical.
I have solved this problem with a script component before, where for each row, if the database id have changed, the connection changes, example below
try {
if (databaseNr != Row.DatabaseNr) {
try {
databaseNr = Row.DatabaseNr;
currentCatalog = "db" + Row.DatabasNr;
connection.ChangeDatabase(currentCatalog);
} catch (Exception e) {
ComponentMetaData.FireWarning(0, ComponentMetaData.Name, e.Message, "", 0);
}
}
string command = "SELECT Id, Name, Surname FROM [" + currentCatalog + "].[TableName] WHERE Id = '" + Row.OrderID + "'";
But it would save me a lot of trouble if this was possible with the lookup component.
So my question is: Is it possible in any way to use column data to change what database to perform a Lookup with the Lookup component?
Grateful for any help!
What you can do is:
Goto control flow
Select your data flow task
Goto properties and select the lookup component
Create an expression for the lookup, you can reuse a query prepared in a script task.
I want to import my IIS logs into SQL for reporting using Bulk Insert, but the comment lines - the ones that start with a # - cause a problem becasue those lines do not have the same number f fields as the data lines.
If I manually deleted the comments, I can perform a bulk insert.
Is there a way to perform a bulk insert while excluding lines based on a match such as : any line that beings with a "#".
Thanks.
The approach I generally use with BULK INSERT and irregular data is to push the incoming data into a temporary staging table with a single VARCHAR(MAX) column.
Once it's in there, I can use more flexible decision-making tools like SQL queries and string functions to decide which rows I want to select out of the staging table and bring into my main tables. This is also helpful because BULK INSERT can be maddeningly cryptic about the why and how of why it fails on a specific file.
The only other option I can think of is using pre-upload scripting to trim comments and other lines that don't fit your tabular criteria before you do your bulk insert.
I recommend using logparser.exe instead. LogParser has some pretty neat capabilities on its own, but it can also be used to format the IIS log to be properly imported by SQL Server.
Microsoft has a tool called "PrepWebLog" http://support.microsoft.com/kb/296093 - which strips-out these hash/pound characters, however I'm running it now (using a PowerShell script for multiple files) and am finding its performance intolerably slow.
I think it'd be faster if I wrote a C# program (or maybe even a macro).
Update: PrepWebLog just crashed on me. I'd avoid it.
Update #2, I looked at PowerShell's Get-Content and Set-Content commands but didn't like the syntax and possible performance. So I wrote this little C# console app:
if (args.Length == 2)
{
string path = args[0];
string outPath = args[1];
Regex hashString = new Regex("^#.+\r\n", RegexOptions.Multiline | RegexOptions.Compiled);
foreach (string file in Directory.GetFiles(path, "*.log"))
{
string data;
using (StreamReader sr = new StreamReader(file))
{
data = sr.ReadToEnd();
}
string output = hashString.Replace(data, string.Empty);
using (StreamWriter sw = new StreamWriter(Path.Combine(outPath, new FileInfo(file).Name), false))
{
sw.Write(output);
}
}
}
else
{
Console.WriteLine("Source and Destination Log Path required or too many arguments");
}
It's pretty quick.
Following up on what PeterX wrote, I modified the application to handle large log files since anything sufficiently large would create an out-of-memory exception. Also, since we're only interested in whether or not the first character of a line starts with a hash, we can just use StartsWith() method on the read operation.
class Program
{
static void Main(string[] args)
{
if (args.Length == 2)
{
string path = args[0];
string outPath = args[1];
string line;
foreach (string file in Directory.GetFiles(path, "*.log"))
{
using (StreamReader sr = new StreamReader(file))
{
using (StreamWriter sw = new StreamWriter(Path.Combine(outPath, new FileInfo(file).Name), false))
{
while ((line = sr.ReadLine()) != null)
{
if(!line.StartsWith("#"))
{
sw.WriteLine(line);
}
}
}
}
}
}
else
{
Console.WriteLine("Source and Destination Log Path required or too many arguments");
}
}
}
I'm building a REST WebService with JAX-RS and Tomcat to consume a MySQL Database.
I'm following this model:
#Path("/login")
public class Login {
String username;
String password;
// This method is called if POST is requested
#POST
#Produces(MediaType.APPLICATION_XML)
public String loginResponseXML(#FormParam("username") String user, #FormParam("password") String pass) {
//Connection to MySQL Database
try {
Class.forName("com.mysql.jdbc.Driver");
Connection con = DriverManager.getConnection("jdbc:mysql://localhost/sakila", "root","larcom");
Statement stmt = con.createStatement();
ResultSet rs = stmt.executeQuery("Select first_name, last_name From actor where first_name='" +
user + "' and last_name='" + pass + "'");
while (rs.next()){
System.out.println(rs.getString("first_name") + " " + rs.getString("last_name"));
username = rs.getString("first_name");
password = rs.getString("last_name");
}
}
catch (ClassNotFoundException e) {
e.printStackTrace();
} catch (SQLException e) {
e.printStackTrace();
}
if (user.equals(username) && pass.equals(password)) {
return ("<?xml version=\"1.0\"?>" + "<auth>200" + "</auth>"); //Success
//return "Success!";
} else {
return ("<?xml version=\"1.0\"?>" + "<auth>404" + "</auth>"); //Damn
//return "Damn!";
}
}
}
I call this method with:
HttpPost httppost = new HttpPost("http://192.168.15.245:8080/org.jersey.andre/rest/login");
Now, my question is:
If I want to query the DB for another table I have to create a new class like Login and make the JDBC connection again?
A new class and a new JDBC connection for each class that make a query to the DB? Performance issues?
Hope you can understand.
Thanks in advance.
A few tips are in order here: Please isolate the DB based code to a "data layer" so to speak...only perform dispatching/business logic within your resource classes.
Now If you are querying a different table, you WILL have a different query! You could either use the same connection (bad) or create a new one and fire a different query(s).
Now whether each resource hits a different table or the same table with a different query depends on your choice of 'representation' for that resource. There is a reason a RDB schema has multiple tables and it's quite common you'll have a different query involving multiple tables or to mutually independent tables.
Performance issues: For 'fresh data' you ARE always going to hit the DB so to speak. If you want to optimize that either develop your own cache (extremely hard) or use approaches like memcached or ehcache to boost performance - before you decide to do that make sure you verify if it's worth it.
Are you going to be having about 1000 DB hits per second? You probably need some performance boosting/handling. Per day...maybe not. Per 2-3 days...YAGNI (You ain't gonna need it, so don't worry for now)
So, for every 'resource' that you design in your application (Login is NOT a resource: See related post: Why is form based authentication NOT considered RESTful?) choose the representation. It may involve different queries etc., for you to return json/xml/xhtml (whatever you choose). Each 'DB related call' should be isolated into it's own 'data layer' - I suggest go with Spring JDBC to make your life easier. It'll take the burden of JDBC plumbing off your shoulders so you can focus on creating your DAOs (Data Access Objects - a patter for Data Access classes. All DAOs logically belong in the data layer)
Hope this helps