I'm creating an app that pulls data from a web server (MySQL), parses it and stores it in a SQLite database using Core Data.
The MySQL database has a 'words' table. Each word can be in a 'category'. So the words table has a field for 'category_id' to join the tables.
I'm having some trouble getting my head around how to replicate this locally in my app. I currently have entities matching the structure of the MySQL database, but no relationships. It seems like in my 'words' entity I shouldn't need the 'category_id' field (I should instead have a one-to-one 'category' relation set-up).
I'm confused as to how to keep this Core Data relationship in sync with the web server?
Assuming you have an Entity for Word and Category you will need to make a relationship (naming may be a bit hazy). Also assuming a Category can have many words and
// Word Entity
Relationship Destination Inverse
category Categories words
// Category Entity
Relationship Destination Inverse
words Word category // To-Many relationship
You are correct you would not need the category_id field as all relationships are managed through the object graph that Core Data maintains. You will still need a primary key like server_id (or similar) in each entity or you will have trouble updating/finding already saved objects.
This is how I deal with syncing data from an external database (I use RESTful interfaces with JSON but that does not really matter)
Grab the feed sorted by server_id
Get the primary keys (server_id) of all the objects in the feed
Perform a fetch using the a predicate like ... #"(serverId IN %#)", primaryKeys
which is sorted by the primary key.
Step through each array. If the fetch result has my record then I update it. If it does not then I insert a new one.
You would need to do this for both Word and Category
Next fetch all objects that form part of a relationship
Use the appropriate methods generated by core data for adding objects. e.g. something like `[myArticle addWords:[NSSet setWithObjects:word1, word2, word3, nil];
It's hard for me to test but this should give you a starting point?
Good to see a fellow Shiny course attendee using stack overflow - it's not just me
Related
I'm using javascript-based CAD configurators to load products (i.e. materials) to be configured into new products (separate table - i.e. widgets). My JSON data columns need to adapt and be valid for materials to be used in client-side configurators for the creation of different kinds of new widgets.
My thought is to add a "data_type" column where each "data_type" is associated with a JSON Schema schematic. This column could be a foreign key or a string as JSON Schemas could be stored in a table (a json "schema" column in a data_types table) or directory (tablename-datatype.schema.json).
At this stage, I'm thinking a data_types table would be more flexible and easier to maintain and serve schemas from. Additionally, this would enable schemas to be shared between tables and I could implement a versioning system to facilitate configurator evolution.
What are the options to implement multiple JSON schemas per column in MySQL? Are there better ways of accomplishing what I'm intending to?
In this use case:
I'm using javascript-based CAD configurators to load products (i.e. materials) to be configured into new products (separate table - i.e. widgets). My JSON data columns need to adapt and be valid for materials to be used in client-side configurators for the creation of different kinds of new widgets.
To facilitate flexible JSON schemas that can be validated and used with relation to individual JSON data columns in multiple tables, the structure I'll implement is:
JSON data is only for client-side processes. This data is related to MySQL entries that have columns and relations that are queried at a higher level. The JSON data requires a flexible architecture for client-side functionality.
Create a datum_schema table. Columns might include key (id), string (name), integer (version), and json (schema) columns. Schemas can be shared, ensure backwards compatibility, and served to multiple client-side technologies.
Tables where entries require a single JSON data record. Create a json column and a reference to a datum_schema record. In the use case, this would be configurators and widgets:
Creating configurators: you create a configurator and datum_schemas records. In the use case, I'll create two schemas: one for settings and one for inputs. Specific settings for loading a configurator instance are stored in a json column within the configurators table.
Configurator table entries store references to their setting and input schemas. User rights will enable some users to create only configurator entries and others to create entries and define schemas.
Creating widgets: as you're using a configurator to create a widget, widget json data will be the input values needed to recreate itself and a reference to its configurator record.
Tables where a single entry may need multiple JSON data records.
Create a separate table to store json data with references to the first table. In the use case, this would be materials.
Creating materials: in the materials table, an entry is made to store any higher level of queryable information (i.e category). In a separate material_data table, entries include a reference to the material, json data, and a reference to the datum_schema. If the material is used in a configurator, the json data will have been structured according to a datum_schema record created by the configurator.
Want to create new kinds of widgets? Define a configurator, the necessary schemas, and relevant material categories.
I believe this to be a reasonable solution but, as I'm new to MySQL and database design in general, I'd appreciate feedback.
I have a JSON file that stores the information about a bunch of recipes, like cuisine, time, the ingredients, instructions, etc. I am supposed to transfer all the data to a MySQL table with the relevant headings.
The "ingredients" and the "instructions" are stored like this:
The instructions and ingredients have several "lines" , stored as a list.
How can we store the ingredients and instructions in a MySQL table, in a line by line format?
something like:
instructions
inst1
inst2
..
The JSON file was created using a python program using the beautiful soup module.
PS: I am very new to both SQL and JSON, so I unfortunately dont have anything to show under "what I tried"...Any help will be appreciated.
Rather than give you the exact answer, I'll give you the process I use to determine a database structure. You're using a relational database, so that's what I'll talk about. Its also good to have a naming convention, I've used CamelCase here but you can do whatever you want.
You mentioned you were using python but this answer is language agnostic.
You've chosen quite a complex example, but I'll assume you understand how to create a table, and use primary keys and foreign keys. If not, maybe you should do something simpler.
Step 1 - Figure out what the entities are
These are the real-life entities which need to represented as database tables. In this case, I'm seeing 4 entities;
Recipe
Keyword
Ingredient
Instruction
Each of these can have a table in MySql. Give them a Primary key which follows a naming convention.
Step 2 - figure out the relationships
It looks like keywords are shared between multiple recipes, so you'll a many to many relationship - this means there's going to be an extra table,
RecipeKeyword
This is just a link between Recipe and keyword to avoid redundancy. It has two foreign keys, RecipeId and KeywordId. At the moment its just a dumb object. In other situations like this, its common for an application to need information about a join - for example, who linked the two things together (consider users, permissions, and a join table with information on who granted the permission)
The other entities are one to many - each will need a foreign key, RecipeId
Step 3 - design each table
As well as having several lists, your Recipe object has some properties. These can be in its table. Most of them are strings in your data, although there are better ways to store things we can keep this simple.
The other entities just have a text field, from your screenshot, only the Recipe has properties.
For this system, you'll need to first insert all Recipe and Keyword objects. There is a common pattern in relational databases where in insert a record, and get its ID so you can insert more stuff which references it.
Step 4 - find a python mysql library
I don't know of one but google will help you find it. The documentation should include the basics of querying.
Step 5 - Insert your data
Here is some psudocode
FOR EACH recipe
INSERT the recipe, and get its ID
FOR EACH keyword
IF the keyword does not exist already
INSERT the new keyword and get its ID
INSERT a record into RecipeKeyword with RecipeId and KeywordId
FOR EACH ingredient
INSERT the ingredient, give it RecipeId as a foreign key
FOR EACH instruction
INSERT the instruction, give it RecipeId as a foreign key
That's it. From here you can select with joins - To form what we're seeing above, you might need to do 3 seperate queries and merge them together into a record object on the python side to reproduce the original structure.
Spring Boot / MySQL / JPA / JPQL
I have an app that I would like to allow for the deletion of an object as long as it isn't being referend by another object (primary key / foreign key relationship)
For the UI I only want to show the delete button if there is no referential integrity issue/constraint (meaning if there is a pk/fk relation I don't want to show a delete button).
Let's say my child object is BookCategory and the parent object is Book. If a BookCategory is assigned to a Book then don't show the delete button, if there is no relationship then show the button.
Obviously in my case, BookCategory object model doesn't have a reference to Book, I could do that... but it seems a bit of resource waste, but maybe if I lazy load it wouldn't.
I'm trying to figure out a way through jpa / #query etc to allow for the UI to know whether to show the delete button or not.
I am looping through BookCategory(s) on the given page, but of course like I say it doesn't know if it is assigned to a book.
Seems like this thing would need to be done all the time, but not sure best how to build it.
Thanks, Keith
Suppose you have many to many connection between the entities.
In your example, you can use a lazy connection to the book and create an entity graph to load it when necessary or use jpql with join fetch.
You have to examine if there is a joined object or not, so you can also write a jpql to count the the joined object.
With Spring Data JPA you can create a composite model with, for example an id list, count... field refer to the books and put the other necessary fields for book category. Then spring data can load this into the model:
#Query(" select new com.mypackage.CustomBookCategory(b.name,count(b.book), ...) from BookCategory b where ...")
List<CustomBookCategory> findByCategory(String category);
I am trying to migrate some data from MySql to the Datastore. I have a table called User with auto-increment primary keys (Bigint(20)). Now I want to move the data from the User table to the datastore.
My plan was let the Datastore generate new Ids for the migrated users and all the new user created after the migration is done. However we have many services (notifications, urls etc) that depend on the old ids. So I want to use the old ids for the migrated user, however how can I guarantee that all new generated ids won't collide with the migrated Ids?
Record the maximum and minimum ids before migrating. Migrate all the sql rows to datastore entities, setting entity.key.id = sql.row.id.
To prevent new datastore ids from colliding with the old ones, always call AllocateIds() to allocate new ids. In C#, the code looks like this:
Key key;
Key incompleteKey = _db.CreateKeyFactory("Task").CreateIncompleteKey();
do
{
key = _db.AllocateId(incompleteKey);
} while (key.Path[0].Id >= minOldId && key.Path[0].Id <= maxOldId);
// Use new key for new entity.
In reality, you are more likely to win the lottery than to see a key collide, so it won't cost anything more to check against the range of old ids.
You cannot hint/tell the Datastore to reserve specific IDs. So, if you manually set IDs when inserting existing data, and later have the Datastore assign an ID, it my pick an ID that you have already used. Depending on the operation you are using (e.g. INSERT or UPSERT), the operation may fail or overwrite the existing entity.
You need to come up with a migration plan to map existing IDs to Datastore IDs. Depending on the number of tables you have and the complexity of relations between them, this could become a time consuming project, but you should still be able to do it.
Let's take a simple example and assume you have two tables:
USER (USER_ID is primary key)
USER_DATA (USER_ID is foreign key)
You could possibly add another column to USER (or another way) to map the USER_ID to DATASTORE_ID. Here, you call Datastore's allocateID method for the Kind you want to use and store the returned ID into the new column.
Now, you can move USER data to Cloud Datastore ignoring the MySQL User ID, instead use the ID from the new column.
To migrate the data from USER_DATA, do a join between the two tables and push the data using datastore ID.
Also, note that using sequential IDs (referred to as monotonically increasing values) could cause performance issues with Datastore. So, you probably want to use IDs that are generated by the Datastore.
I can find all over the web articles to migrate an existing MySQL database to a SimpleDB or CouchDB or what so ever.
But If I go ahead and migrate my millions of records to Riak or CouchDB, is there any way to migrate back?
Would it be possible in theory to regenerate all the relations between my data that has no clear indexes to a relational database and regenerate all the relations?
Example:
I have two tables in MySQL one is person and one is job, job is related to person - in one to one relationship.
If I switch that to couchDB, person document will have the job defined as text value for the key "job"
inserting this relation in couchDB is very simple from MySQL. Select.. combine then dump one person object.
Now, split one person object into two objects - and find the correct key for the certain job. How?
Any help is appreciated!
Here's my idea:
For the example, I'd start with looping through all records to create a list of jobs, then adding those to the jobs table. Then do a loop again, this time matching the 'job' in persons to the newly created job in mysql and fetching the id, adding that id in the person table 'jobid' foreign key field.
This way of analysing the data could probably be generalised in a script, but you would need to present it a 'translation' array or something, defining the relations.
To answer your question, yes, in theory it should be possible.