Are there any alternative options to Confluent's MySQL sink connector?
The JDBC sink connector is open source and requires no license to use
What are you looking to do with the MySQL sink connector? I'm part of a team working on an alternative option to Kafka Connect so this might fit what you're looking for.
Check out: GitHub - ConduitIO/conduit: Data Integration for Production Data Stores.
Conduit is meant to be a bit more general-purpose than connect. Conduit has the capability of being a drop-in replacement for Connect and we're working hard to make that even easier. We're still very much in the early stages of this project and we're trying to build more and more connectors. Our philosophy around connectors is to be real-time first, double down on change data capture capabilities, and have permissive licensing. If you need something that isn't there, I'd recommend jumping into the github discussions on the repo, the team is pretty responsive. or join our discord to speak directly to the team for questions or feedback!
Related
I am currently looking for a version control tool for MySQL DB which is cloud hosted. The new application/tools needs to be cloud hosted as well.
I am interested in something which is free or have nominal charges. I looked over the internet but could not find a satisfactory solution. My requirement is very simple with the tools i.e. it should just capture all the changes to the DB and if required I can switch over back to the previous state of the DB.
Can you please advise some options which i can explore.
Thanks
There are a couple of tasks/tools you need to implement to consider the minimal DB version control support.
Source Control Repository - tool to physically keep track of changes made to files (i.e. git, SVN, TFS, etc.). Git is free.
Extract a representation of the DB - the DB needs to be represented in some kind of file format in order to be version controlled. It is not recommended that the database files themselves be version controlled, nor is it recommended to take backups of the db and storing those. You'll want to store the DB in a semantic representation, such as .sql file(s). There are a couple free tools that can be used (i.e. mysqldump, HeidiSQL). I like using HeidiSQL as it has a feature that allows you to selectively create dump scripts. These script can be stored in the version control system.
If you are looking at automation, you'll want to consider implementing a build server tool like Jenkins. Jenkins can be configured to poll a git repository and then execute build scripts (created by you) if it notices a change.
Some of my answer is subjective and I have glossed-over several intricacies of DB Version Control. You can research more questions about DB Version Control that have some pretty good answers and direction to well-written blog posts. There are also a plethora of paid tools out there that give significant DB Version Control functionality. To name a few, there's Red Gate, DBMaestro, Liquibase, RoundhousE, MySQL Workbench, and others.
I hope this helps.
I've been running in circles looking for a way to deliver data from a proprietary flat file database (based on the ProvideX platform) to a client-thick web application that makes RESTful requests and expects JSON responses.
ProvideX and Sage MAS 90 provided an ODBC driver that works for pulling tables, but I can't think of a good way to connect the dots without needing to program a bunch of server-side code.
Before I go down the path of programming custom server-side middleware, does anyone have any bright ideas, (or obvious ideas that I have overlooked)?
I am not locked into any particular architecture at the moment because we are hashing out requirements for the web application, so any ideas would be helpful.
ProvideX/Sage provides a web services module, but I can't use it because my company has refused to invest in the software module and upgrade costs. Let's not let that be a distraction, however, because I am still looking for a way to use the ODBC driver in this question thread.
ODBC-ODBC Bridges exist but all the ones I know are commercial.
I'm trying to get all the historic information about a sensor of Fi Ware.
I've seen that Orion uses Cygnus to store historics in Cosmos. Is that information accesible or is it only possible to use IDAS to get it?
Where could I get more info about this?
The way you can consume the data is, in an incremental approach from the learning curve point of view:
Working with the raw data, either "locally" (i.e. logging into the Head Node of the cluster) by using the Hadoop commands, either "remotely" by using the WebHDFS/HttpFS REST API. Please observe within this approach you have to implement whichever analyzing logic you need, since Cosmos only allows you to manage, as said, raw data.
Working with Hive in order to query the data in a SQL-like approach. Again, you can do it locally by invoking the Hive CLI, or remotely by implementing your own Hive client in Java (there are some other languages) using the Hive libraries.
Working with MapReduce (MR) in order to implement strong analysis. In order to do this, you'll have to create your own MR-based application (typically in Java) and run it locally. Once you are done with the local run of the MR app, you can go with Oozie, which allows you to run such MR apps in a remote way.
My advice is you start with Hive (the step 1 is easy but does not provide any analyzing capabilities), first locally trying to execute some Hive queries, then remotely implementing your own client. If this kind of analysis is not enough for you, then move to MapReduce and Oozie.
All the documentation regarding Cosmos can be found in the FI-WARE Catalogue of enablers. Within this documentation, I would highlight:
Quick Start for Programmers.
User and Programmer Guide (functionality described in sections 2.1 and 2.2 is not currently available in FI-LAB).
Back in the MSSQL 2000 timeline, there was an IIS integration layer that allowed HTTP GET commands to make select statements, and there were other SqlXml niceties that worked (not that fast or well but they worked) out of the box. I gave a chance to expose database stuff fairly quickly.
What is the comparable technology for MSSQL 2008/2012? I saw slashDb (http://www.slashdb.com/) and it seems to do that, but I am trying to understand the other options out there. Just SQL Server crud and sproc access.
Thanks.
Yes, SlashDB does exactly that and more. Full disclosure: I am the founder and CEO.
Once SlashDB is installed you would use its web interface to connect it with your database. Depending which database login and database schema you use for that connection, you will have the tables and views from that schema turned into of URL endpoints.
Those URLs can be followed in the browser but they are also API endpoints in JSON, XML or CSV. It works for reading and writing (you can control that in user configuration).
In addition to that you can define a set of parameterized SQL queries. Each query is given a name and instantly becomes an API endpoint too.
In order to help you getting started easily SlashDB is available on AWS and Azure marketplaces, as a Docker container from DockerHub, pre-built virtual machines or as .rpm and .deb packages for installation directly on Linux.
For more technical info please visit: https://docs.slashdb.com
The nearest equivalent may be SOAP/HTTP endpoints, however Microsoft has deprecated them for various reasons and recommends WCF or ASP.NET instead. Although the simplest way to get a quick CRUD setup is probably to use a framework or ORM that generates it for you, like LINQ to SQL or whatever else suits your needs.
I'm looking for an ETL solution that we can create a configure by hand and then deploy to run autonomously. This is basic transformation, it need not be feature heavy. Key points would be free or open source'ed software that could be tailored more to suit specific needs.
In fact, this could be reduced to a simple DB migration tool that will run on a Linux server. Essentially the same as the above but we probably won't need to validate / transform the data at all besides renaming columns.
I forgot to mention that this is going to have to be very cross platform. I'd like to be able to deploy it to a server, as well as test it on OSX and Windows.
Try Pentaho or Talend. Pentaho has a nice job-scheduling paradigm as well as the ETL workbench (Kettle). I haven't used Talend, but I've heard good things and I imagine it carries similar functionality.