I have series of documents that I need to migrate into MarkLogic. The documents are available to me via RESTful services in JSON. What I want to know is there anyway, such as through the MLCP or Query Console to call those RESTful services and pull in the data, otherwise I have to create a small Java app and dump the files to a share then pick them up from MarkLogic.
mlcp is designed to source data from the file system or a MarkLogic database. Take a look at the Java Client API to perform ingestion from other sources. For example, you can fire up your favorite HTTP client in Java and add the results to a DocumentWriteSet. The write set acts like a buffer, allowing you to batch requests to MarkLogic for efficiency. You can then send that DocumentWriteSet to MarkLogic with one of the DocumentManager.write() methods. Take a look at the documentation for many more details or the "Bulk Writes" section of the getting started cookbook.
Related
Lets say you have a business Application producing and storing Enriched Product Master Data in its own environment, one the enrichment is completed you want to make that data available on a CouchBase Database.
In order to get that data from Business Application's environment into CouchBase, let's assume I want to use Kafka to broadcast the changes and Nifi to distribute it to the final desination (CouchBase).
But CouchBase takes JSON format files. Can I use Kafka or Nifi to convert the pulled data into a JSON format? I know I can for instance put solution such as Attunity between the Business Application and Kafka to replicate the data real time. But let us assume that there is no budget to implement the solution attunity, so one will temporarily use a REST API on the Business Application side and pull that data with (based on made changes) Kafka, can I convert the data into JSON with Kafka? or NiFi?
EDIT:
Well the reason why I want to know if NiFi can do this, is because our landscape is a bit more complex than I described. Because between CouchBase and the Business Application, you have:
[Business App] - [ X ] - [Kafka] - [NiFi] - [DC/OS with KONG API Layer] - [CouchBase Cluster].
And I want to know if I should implement a new solution for Data Replication on the spot of the X, or should I just make use of the Business App REST API and pull data from the REST API with Kafka and Convert my data to JSON in NiFI.
There is a Couchbase sink for Kafka Connect. This will enable you to do exactly what you want. Simple configuration-file based approach.
I'm creating a multi-platform app, mostly for web interface, mobile and a windows application. The app will manage user task lists and sync them to the server, but also store them locally for processing data faster.
My idea of architecture until now is:
Keeping most of the processing on client side, eventually syncing with the server.
Developing an API to provide and receive data that will be saved on the server (basically just a json wrapper web service)
The data flow:
user Authenticates -> Requests updated Json objects to the server -> populate client-side objects -> work with client-side objects -> send a json object back to the server -> server updates data.
Is this a good approach? I've never done this, can you guys give me some tips?
I think you are on the right track. The idea is to decouple the front-end from the back-end. The backend shall expose a set of CRUD (Create, Read, Update, Delete) functions as RESTful JSON web services. All your different flavours of UI (mobile, web, windows) can consume the same API.
I would recommend for the web front-end to take a look at AngularJS together with bootstrap.
Regarding the backend, you could implement it as a simple Java web application with Jersey/JAX-RS or alternatively, you could check Node.js + Express.
I want to expose data in a mysql data source as a soap web service using Talend. How can I achieve that. Please advice.
The most powerful way to do so to do so is probably to build a WSDL using Talend Open ESB Studio (Tutorial)
Alternatively, if you don't like to use another piece of software or you prefer a quick-and-dirty solution, you can achieve this with standard Talend Open Studio for Data Integration. You just need to draw a standard Talend job which redirects your mysql data output to a tBufferOutput instance. Then, you must export your job as Axis webservice, finally deploy it in your application server (Tutorial). This way, your buffer data will be automatically exposed via web service. But it has a drawback: you cannot tune the response (ie, add labels to fields or refactor the SOAP response structure at all).
Has anyone had much experience with data migration into and out of NetSuite? I have to export DB2 tables into MySQL, manipulate data, and then export ina CSV file. Then take a CSV file of accounts and manipulate the data again for accounts to match up from our old system to new. Anyone tried to do this in MySQL?
A couple of options:
Invest in a data transformation tool that connects to NetSuite and DB2 or MySQL. Look at Dell Boomi, IBM Cast Iron, etc. These tools allow you to connect to both systems, define the data to be extracted, perform data transformation functions and mappings and do all the inserts/updates or whatever you need to do.
For MySQL to NetSuite, php scripts can be written to access MySQL and NetSuite. On the NetSuite side, you can either do SOAP web services, or you can write custom REST APIs within NetSuite. SOAP is probably a bit slower than REST, but with REST, you have to write the API yourself (server side JavaScript - it's not hard, but there's a learning curve).
Hope this helps.
I'm an IBM i programmer; try CPYTOIMPF to create a pretty generic CSV file. I'll go to a stream file - if you have NetServer running you can map a network drive to the IFS directory or you can use FTP to get the CSV file from the IFS to another machine in your network.
Try Adeptia's Netsuite integration tool to perform ETL. You can also try Pentaho ETL for this (As far as I know Celigo's Netsuite connector is built upon Pentaho). Also Jitterbit does have an extension for Netsuite.
We primarily have 2 options to pump data into NS:
i)SuiteTalk ---> Using which we can have SOAP based transformations.There are 2 versions of SuiteTalk synchronous and asynchronous.
Typical tools like Boomi/Mule/Jitterbit use synchronous SuiteTalk to pump data into NS.They also have decent editors to help you do mapping.
ii)RESTlets ---> which are typical REST based architures by NS can also be used but you may have to write external brokers to communicate with them.
Depending on your need you can have whatever you need.IN most of the cases you will be using SuiteTalk to bring in data to Netsuite.
Hope this helps ...
We just got done doing this. We used an iPAAS platform called Jitterbit (similar to Dell Boomi). It can connect to mySql and to NetSuite and you can do transformations in the tool. I have been really impressed with the platform overall so far
There are different approaches, I like the following to process a batch job:
To import data to Netsuite:
Export CSV from old system and place it in Netsuite's a File Cabinet folder (Use a RESTlet or Webservices for this).
Run a scheduled script to load the files in the folder and update the records.
Don't forget to handle errors. Ways to handle errors: send email, create custom record, log to file or write to record
Once the file has been processed move the file to another folder or delete it.
To export data out of Netsuite:
Gather data and export to a CSV (You can use a saved search or similar)
Place CSV in File Cabinet folder.
From external server call webservices or RESTlet to grab new CSV files in the folder.
Process file.
Handle errors.
Call webservices or RESTlet to move CSV File or Delete.
You can also use Pentaho Data Integration, its free and the learning curve is not that difficult. I took this course and I was able to play around with the tool within a couple of hours.
We would like to hook up an iPhone app directly to a database in the cloud. From the iPhone app, we would like to use some REST API to retrieve data from the cloud database. We prefer using MySQL or Mongo as the data store.
We prefer not setting up our database system on AWS or another hosted provider. We would like a cloud service that offers the database layer out of the box.
What database-as-a-service companies support this, and what are the pros/cons of each?
We looked around, and it seems like MongoLab is the only service offering a REST API.
Mongo is a great database for API interaction as the query language uses javascript expressions. Here are the open source API libraries that are implemented in Python
Sleepy Mongoose
Eve
Primary advantage
JavaScript can handle the querying part of data retrieval
Primary disadvantage
The API libraries above have difficulty capturing complex queries