Migrating from MySQL to Couchbase Server (NoSQL) - mysql

I am planning to move some of my RDBMS tables to a NoSQL environment. I have different modules and associated tables as shown below.
1. General
News table
Trainings table
Knowledge table
2. Application_x
Clicks table
Crashes table
Note : This is a sample scenario
In this case, this is how i defined Nosql (Couchbase server) structure. Please correct me if this is wrong.
The purpose of migration is mainly intended for searching.
{
"General":[
"news" : [{
"id" : "123",
"title" : "test title"
},
{
"id" : "345",
"title" : "test title 2"
}],
"trainings" :[{
"id" : "1",
"name" : "training 1"
},
{
"id" : "2",
"name" : "training 2"
}],
"knowledge" :[{
"id" : "1",
"categ" : "programming"
},
{
"id" : "2",
"categ" : "management"
}]
],
"Application_x": [
"clicks" : [{
"userid" : "1",
"area" : "1850",
},
{
"userid" : "2",
"area" : "258",
}],
"crashes" :[{
"userid" : "1",
"severity" : "1",
},
{
"userid" : "2",
"severity" : "8",
}]
]
}
Can someone correct me if my approach is not correct ?
Thanks in advance,
Tismon Varghese.

By reading your question, I am left scratching my head as to what your approach is. In a NoSQL database (such as couchbase) - you don't have the idea of tables and columns. Each object is serialized to JSON and stored in plain text. Yes, this creates duplication, but the drawbacks of duplication are greatly outweighed by the benefits of scalability.
In this example, using Couchbase, you would probably want to create one Couchbase bucket per application. That way, should you need to migrate to a different Couchbase cluster at a later date, a minimal amount of configuration is required. Each row in your tables gets created as a separate object in Couchbase. There is no need to separate the object types within the bucket.
This blog entry contains some detailed instructions on how to migrate from mySQL to Couchbase.
On a side note, I might recommend using Couchbase combined with Elasticsearch.

Related

How to generate all test cases from a JSON schema

I am developing a system where we create documents that are created from json-files. The json files are described by a json schema of the following kind, where a key either could have a static default value or have one of multiple enumerated values:
{
"name" : { "default" : "John Doe" },
"bilingual" : { "type" : "boolean" },
"kind_of_document" : {
"type" : "integer",
"enum" : [0, 1, 2]
}
}
What I want to do is to create test files for all possible combinations of values from the schema. In the above case there would be six different json files:
{ "name" : "John Doe", "bilingual" : true, "kind_of_document" : 0 }
{ "name" : "John Doe", "bilingual" : true, "kind_of_document" : 1 }
{ "name" : "John Doe", "bilingual" : true, "kind_of_document" : 2 }
{ "name" : "John Doe", "bilingual" : false, "kind_of_document" : 0 }
{ "name" : "John Doe", "bilingual" : false, "kind_of_document" : 1 }
{ "name" : "John Doe", "bilingual" : false, "kind_of_document" : 2 }
This is a simplified version; in a normal case there may be 20-30 keys, of which most have default values, but there may still 5-10 keys that can have multiple values. They number of keys may vary from one document to another.
The problem:
How do I iterate over the list of keys to generate all possible combinations of values? I suppose that some form of recursion is the way to go, but I can't figure out what the algorithm should be. It seems to me that this should be a fairly common problem, so somebody has most likely solved it before.
I have tried googling for it, but I don't know how to formulate the query to avoid just general questions about generating combinations of list elements, such as
How to get all possible combinations of a list’s elements?
I have also looked at itertools, but I am not sure how it could me solve the problem:
https://docs.python.org/3/library/itertools.html#itertools.permutations
I am writing this in python, but of course the general algorithm for solving this would be language independent.

Custom json in Drupal 7

I am a beginner in Drupal 7 and I am trying to build a custom JSON by using the taxonomy terms. I have implemented the taxonomy terms with parent child relationship and I want to build the same for json. For example, I have a term term Activism which a parent for several other term. Thus, I am looking for a JSON something like this.
{
"name":"Terms",
"children" : [
{
"name":"Activism",
"children" : [{
"name" : "American Civil Liberties Union (ACLU)",
"size" : "1"
},
{
"name" : "Bertrand Russell",
"size" : "1"
}]
}]
}
Can anyone help how to proceed with it. It seems the JSON data document does not provide such relational structuring.

Translating engine gives same guids & viewableIDs to diferrent files of complex model

Recently I was trying to implement scenario where I could link views in 3D viewer with views on original revit model.
I zipped linked files and upload them
Upload more times same zip file with changed rootFileName to provide all aspects of a model and translate all
download their manifest.jsons
link views on revit side with view in the viewer using viewableID and uniqueID of original revit view
In here I will provide real shortened examples of manifest outputs
eg file 1.
{
"guid" : "578dd62c-d5dc-8da0-1c75-aa6b0be8ba76",
"name" : "3D pohled",
"success" : "100%",
"hasThumbnail" : "true",
"status" : "success",
"type" : "folder",
"progress" : "complete",
"children" : [{
"guid" : "07e30540-b182-cd58-1c1f-eb46bc58fb9c",
"name" : "{3D}",
"success" : "100%",
"size" : 771479.0,
"hasThumbnail" : "true",
"role" : "3d",
"viewableID" : "8af1141e-a7ca-22b4-d104-d7331a1c0454",
"status" : "success",
"type" : "geometry",
"progress" : "complete",
"children" : [{
"guid" : "429ba882-f0a0-40fa-96e6-c6e02d9fc601-00022cda",
"name" : "{3D}",
"role" : "3d",
"camera" : [22586.240234, 40687.59375, 32810.621094, 54.133858, -29.527559, 16437.923828, -0.160696, -0.29039, 0.943319, 0.547078, 0, 1, 1],
"status" : "success",
"type" : "view",
"progress" : "complete"
}, {
"guid" : "9a3b544b-3c15-9304-f6f8-2c4db7a7dcb3",
"type" : "resource",
"role" : "graphics",
"urn" : "urn:adsk.viewing:fs.file:dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6dGVzdGJfMTIzZWUvRkhfRmFzJUMzJUExZGFfdi4wMS56aXA/output/Resource/3D_pohled/_3D_/_3D_.svf",
"mime" : "application/autodesk-svf",
"size" : 680482.0
}
]
}
]
}
eg. file 2
{
"guid" : "578dd62c-d5dc-8da0-1c75-aa6b0be8ba76",
"name" : "3D pohled",
"success" : "100%",
"hasThumbnail" : "true",
"status" : "success",
"type" : "folder",
"progress" : "complete",
"children" : [{
"guid" : "07e30540-b182-cd58-1c1f-eb46bc58fb9c",
"name" : "{3D}",
"success" : "100%",
"size" : 7648740.0,
"hasThumbnail" : "true",
"role" : "3d",
"viewableID" : "8af1141e-a7ca-22b4-d104-d7331a1c0454",
"status" : "success",
"type" : "geometry",
"progress" : "complete",
"children" : [{
"guid" : "429ba882-f0a0-40fa-96e6-c6e02d9fc601-00022cda",
"name" : "{3D}",
"role" : "3d",
"camera" : [40927.800781, 19529.832031, 36075.765625, 54.958988, -27.723097, 16418.119141, -0.359014, -0.171787, 0.917387, 0.320998, 0, 1, 1],
"status" : "success",
"type" : "view",
"progress" : "complete"
}, {
"guid" : "9a3b544b-3c15-9304-f6f8-2c4db7a7dcb3",
"type" : "resource",
"role" : "graphics",
"urn" : "urn:adsk.viewing:fs.file:dXJuOmFkc2sub2JqZWN0czpvcy5vYmplY3Q6dGVzdGJfMTIzZWUvRkhfVFpCX1ZaVC56aXA/output/Resource/3D_pohled/_3D_/_3D_.svf",
"mime" : "application/autodesk-svf",
"size" : 7552662.0
}
]
}
]
}
as you see each time URN of file is different, still viewableID and guids are the same. This is not suppose to happen and represents a real problem when working with the information. In manifest.json is also missing information about original revit guid(uniqueid) from which view it was generated, which is needed to uniqely identify view on revit side.
I think it is a bug and needs to be solved and also adding revit view guid into output manifest would help a lot. Because name property is not unique enough, the other guids has no practical use.
Any help would be greatly appreciated
The ids will be the same if they are translated from the same file (i.e. the unique ID used by Revit on its views), regardless the file name. So if you have a .rvt with a view, duplicate this file with different name, translate them both, they will have the same id on that view.
EDIT October 17, 2016
We have updated the translation engine in 9/26/2016 and the viewableID in the bubble is the same as the UniqueId of a Revit View, for both 3d and 2d. That way you can use the viewableId to implement your feature feature.
It is reasonably well known that Revit elements' "unique" ids are not, in fact, true GUIDs. Two copies of a file will have exactly the same "unique" ids for the elements in those files (they are not re-generated on a save-as operation). The purpose of the Unique id in Revit was to provide a mechanism to facilitate worksharing where element ids are not "stable" and may change during a reload latest event where element ids from the central conflict with local Element Ids.
Here is some information from The Building Coder about this topic:
http://thebuildingcoder.typepad.com/blog/2014/04/element-id-export-unique-navisworks-and-other-ids.html#3
In terms of your specific question: "How can i identify, uniquely, the views in revit when traversing manifest.json?" I would suggest that you might (if you have the ability) add the Revit filename as part of the name of the exported view which you have uploaded and translated. Tease that back out of the URN to determine which view you actually have.
If that doesn't work, maybe you could help us better understand this statement:
Upload more times same zip file with changed rootFileName to provide all aspects of a model and translate all
what is this attempting to achieve? different implementations on the viewer side of the same view data? And if so, maybe you can tell us more about exactly what you're trying to ultimately achieve and we can offer some more specific suggestions.

Deeply nested JSON documents in Apache Solr

I have a deeply nested document(pseudo structure as shown below):
[{
"id": "1",
"company_id": "1",
"company_name": "company_1",
"departments":[{
"dep1" : [{
"id" : 40,
"name" : xyz
},
{
"id" : 41,
"name" : xyr
}],
"dep2": [{
}]
}]
"employeePrograms" :[{
}]
}]
How can I index these type of documents in Apache Solr?
Documentation gives the idea of immediate child documents alone.
Unfortunatelly i'm don't have huge experience with this technology, but want to help. Here is some official documentation, that might be useful: oficial doc
more specific
If you have some uncommon issue, tell about it, maybe any error, or whatever.. I would try my best to help)
Upd1 :
Solr can only maintain a 'flat' representation of the data. What you weretrying to do is not really possible. There are a number of workarounds, such as using dynamic fields and using a solr join to link multiple data sets.
Speking about a deep nesting ? I've found such an example of work around.
If you had something like that:
"docs": [
{
"name": "Product Name",
"categories": [
{
"name": "Category 1",
"priority": 8
},
{
"name": "Category 2",
"priority": 6
}
...
]
},
You have to modify it like that to make it not deeply nested :
"docs": [
{
name: "Sample Product"
categories: [
{
priority_category: "9_Category 1",
},
{
priority_category: "5_Category 2",
}
...
]
},
So, you've done something similar, check if there are any errors anywhere

Is there any way to preserve the order while generating patch for json files ?

I am new to Json stuff i.e. JSON PATCH.
I have scenario where I need to figure out between two version of Json files of same object, for that I am using json-patch-master.
But unfortunately the patch generated interpreting it differently i.e. the order differently hence getting unexpected/invalid results.
Could anyone help me how to preserve the order while generating Json Patch ?
**Here is the actual example.
Original Json file :**
[ {
"name" : "name1",
"roolNo" : "1"
}, {
"name" : "name2",
"roolNo" : "2"
}, {
"name" : "name3",
"roolNo" : "3"
}, {
"name" : "name4",
"roolNo" : "4"
} ]
**Modified/New Json file: i.e. removed 2nd node of original file.**
[ {
"name" : "name1",
"roolNo" : "1"
}, {
"name" : "name3",
"roolNo" : "3"
}, {
"name" : "name4",
"roolNo" : "4"
} ]
**Patch/Diff Generated :**
[ {"op":"remove","path":"/3"},
{"op":"replace","path":"/1/name","value":"name3"},
{"op":"replace","path":"/1/roolNo","value":"3"},
{"op":"replace","path":"/2/name","value":"name4"},
{"op":"replace","path":"/2/roolNo","value":"4"}]
Very time I generate Diff/Patch it is giving different path/diff results.
And moreover the interpretation is different i.e. order is not preserving.
**Is there any way to get expected results i.e. [ {"op":"remove","path":"/1"} ] , in other words generated a patch/diff based some order so will get what is expected. ?
How to handle this kind of scenario ?**
Please help me.
Thank you so much.
~Shyam
We are currently working on this issue in Starcounter-Jack/JSON-Patch.
It seems to work nice with native Array.Observe- http://jsfiddle.net/tomalec/p4s7aw96/.
Try Starcounter-Jack/JSON-Patch issues/65_ArrayObserve branch
we will release it as new version once shim and performance will be checked.
Feel free to add you comments at JSON-Patch issue board