I have recently pulled multiple JSON files from a SQL database and I would like to load them into my Google Datastore. Can anyone suggest the best way to go about this. I have read the docs and they detail how to create entities but I cannot determine how to do a bulk data load. Any tips or tricks would be welcome.
Two years later and no answer! The key to doing this right now, in 2017, seems to be the new Dataflow thing in Google Cloud. There are SDKs for both Java and Python, but it's still so new I'm using the Java SDK, the 1.9 version. I've adapted two of the examples and have it putting data into the Datastore. It seems to play nice with namespaces so far, but it's a little difficult to make fields with parent/child relationships.
Related
With a colleague of mine, we're building an app written in Dart with Flutter on Android Studio. We've arrived at the point where we need to start integrating a database to collect and send user filled data, and so we chose MongoDB which will be integrated into Docker so that our app is ready to function on multiple devices. Since we will have many users and each of them will be entering their own data, we have a lot of parameters to take into account so we're creating a JSON skeleton to map out the structure of what data goes where. The obstacle is we have no clue what the best way is to approach MongoDB-Docker integration with our Android Studio code, as it is our first time using MongoDB and Docker. Any good tips or resources that could put us on the right track ? Thank you
Hi your real question should be : what will you use in beetween those two ? You should (I guess) create an API to simplify and securise your user-DB interactions.
If you're not familiar with those principles a quick reasearch should help.
If you want to continue without any API, you should put many effort into having a VERY clean code as your app will have a lot of information inside its source code and proceed to create a documentation. Also you could use an ORM.
I am working on a project with a large CSV file that containes the location and movement of users. I would like to place this on a custom map in Google Maps via bluemix and use Bluemix Services to explore the data.
The primary goals are:
Getting the CSV data on the custom Google map. When running, the data should progress in time and show the movement of users.
Making the CSV points cluster for UX. (so that points that are near each other would stack together)
My primary question is how to get started on this. Do you reccommend i work on this locally and then connect Bluemix to my project or can i create all of this in Bluemix. I would much prefer the last option if possible.
If you have any suggestions to Watson Services or other Bluemix Features that may improve the app this is also greatly appreciated
Thank you for your time.
Ps. I realize Google Maps integrates best with Java Script. Do you recommend converting the CSV to Json when working with Bluemix?
This is a quite broad question and maybe not well suited for stackoverflow (stackoverflow is not a discussion forum, it is a competition of what answer is the most accurate one for a very strict question) so maybe this is a question for https://developer.ibm.com/answers/ instead.
That said, bluemix works well with js, java, ruby, python, go, php, etc (probably js and java better than the others) so I'd go in this direction. Also, I think you should investigate bluemix geospatial analytics (https://console.ng.bluemix.net/catalog/services/geospatial-analytics/) for your application.
For data storage, I suggest you to take a look on cloudant (https://console.ng.bluemix.net/catalog/services/cloudant-nosql-db/) which is a very popular option in bluemix and suits well for most cloud apps. If you want to take a more traditional approach, you can also consider a relational db such as DB2 (https://console.ng.bluemix.net/catalog/services/ibm-db2-on-cloud/)
I'm new to webdev and I'm trying to use passport for registration/authentication on a site I'm setting up. I'm also going to write an application in node later on that will be using some of the user data (users will need to provide an API key for an account on another site that I will use to pull data into the application).
At the moment, the main issue I'm having is figuring out what goes where. I've found plenty of resources that explain how to create an app using passport, but nothing shows how it would be incorporated into your website or where the files should be in relation to your website. I'm relatively new to Node.js, and while I've written a few small applications I have never hosted them anywhere.
Bonus question: I'm using MongoDB with passport and I was also planning to use it to store some JSON my application will be receiving from API calls. However, I wanted to use MySQL to store some data as well. More specifically, I'm planning to save the raw JSON then I'll create a relational database out of the data I need from the JSON and then keep the rest in MongoDB for easy access. Is this common/smart, or should I focus on keeping everything in my MongoDB? I'm relatively new to NoSQL.
Thanks in advance for any help.
I would reference this tutorial. I just recently used this to help myself with a new application. Also there is an example of the same thing but in SQL here. So not sure what you mean by " where the files should be in relation to your website". The information related to to authentication should go in your database.
To your "bonus question" you can use two databases. The key here is to ask yourself why and what are the true needs for data, and how is this data accessed and used. From ground up I would like one and stick with it. If at some point later you realize a certain type of data would be better in a different database then you can add it.
Side note: look into an IDE such as webstorm to help you out.
I am working on my first iOS-application and I want to use MySQL as my remote database.
I've been googing around and reading here at Stack, but I can't find my new answers to the question. Does there exists any good wrappers out there? I found this link: mysql for ios, but it clearly states that it is not guaranteed to be accepted in the App Store.
I really want to have a wrapper rather than using some sort of webservice.
Anyone have some more updated news on this?
Your best bet is to use SQLite or CoreData libraries, they are very low overhead. CoreData is built into iOS functionality, SQLite just needs the .db file and a library (part of iOS) imported.
If you could somehow get your MySQL database online and expose it via REST you could possibly use REST to get and set data into and out of the database. But this will slow your app to a crawl.
CoreData is fast, and when using the data in context like that, it simplifies everything. Writing SQL statements is slowly becoming an archaic process.
In this thread someone pointed me to use sqlalchemy-migrate to help with a fast-changing web application using sqlalchemy.
However a Do It Yourself method was also recommended consisting in manually writing CSV columns for the new database schema, and finally import them.
The problem is that I can't find real-world examples of sqlalchemy-migrate. Ressources that I have found at best decribe adding a single column or a column rename. The official documentation essentially describes the API and it's hard to see how to use migrate effectively. From the doc I cannot even know if migrate could help changing the database engine, from sqlite to mysql for example, while the DIY solution would to the job.
I really want to see code that would make some non-trivial transformations of a database schema and proving that migrate is really a useful tool.
Where can I find good examples/tutorials for sqlalchemy-migrate ?
Thanks !
Don't forget about google code search when looking for real work examples of code. For instance the follow search:
http://www.google.com/codesearch?hl=en&lr=&q=%22from+migrate+import%22+lang:python&sbtn=Search
Will pull up a number of real migration scripts. It basically looks for Python files with "from migrate import" in the file.
Work through some of these and see if you can follow what they're doing.