I have to do the following steps two or three times a day
log in into Elasticsearch
Go to Dev Tools
Run a specific query by selecting it and pressing ctrl + enter
Query that I have to run
Select the results that returns in the "buckets" and copy it.
The yellow markdown in the image is what I have to select and copy
Then I go to https://www.convertcsv.com/json-to-csv.htm and paste the results so it converts to CSV.
Where I have to paste the results.
I can then download the CSV and then import it into google sheets so I can view the results in a Looker Dashboard.
Button to download the converted CSV.
This take me some time everyday and I would like if there is any way that I could automate such routine.
Maybe some ETL tool that can perform at least part of the process or may some more specific way to do it with python.
Thanks in advance.
I don't have much experience with what I want to do and I tried to search online similar issues, but couldn't really find anything useful.
I don't know you tried, but there is a reporting tool on elasticsearch inside the "Stack Management > Reporting". On the other side, there are another tools which you can work from a server with crontab. Here are some of them :
A little bit old but I think it can work for you. ES2CSV. YOu can check there are examples inside the docs folder. YOu can send queries via file and report to CSV.
Another option which is my preference too. You can use pandas library of python. YOu can write a script according to this article, and you can get a csv export CSV. The article I mentioned is really explained in a great way.
Another alternative a library written in Java. But the documentation is a little bit weak.
Another alternative for python library can be elasticsearch-tocsv. This one is a little bit recently updated when I compare it to first alternative. But the query samples are a little bit weak. But there is a detailed article. You can check it.
You can use elasticdump, which is written on NodeJS and a great tool to report data from elasticsearch. And there is a CSV export option. You can see examples on GitHub page.
I will try to find more and I will update this answer time by time. Thanks!
Related
I am new to mysql and databases in general. I've come across a problem that I can't seem to find a solution too. I'm not sure even how to word the scenario.
Basically I have a concatenated field that I need to take "explode" and make each value its of that field its own row. I'm pulling this data from a daily feed, so this needs to be automated as either a trigger, stored procedure, or scheduled job in Talend. Here is a simplified version of the situation. Please see the "current file format" and the "desired outcome" pics. Sorry- this is my first post here! Any help would be appreciated!
I am using a mysql database. Currently the file is coming in via csv export, but I will be connecting via Talend Open Studio to sync the data. See the links below for and example. Here is a sample.
A precise answer is difficult without seeing your code, and the sample link returns 404 for me. You should be able to do this within Talend. Try using Talend’s tExtractDelimitedFields or tExtractRegexFields commands to achieve the result you want.
I need to create a daily process which pulls market data from a website http://www.apxendex.com/index.php?id=137.
I had originally created an excel+VBA sheet which added the data to an xml file and this worked fine. However, the machine which I will be putting the code on doesn't have excel so all my work was pointless (stupid I know).
I'm not looking for someone to write anything for me, but some general tips on where to start, what language to go for etc. would be helpful.
Thanks
How did you get your information from the website?
There is a DataCapture API for that site. You can find the manual on this site (you have to login to see the manual). I guess it would be the easiest way to go the recommended way.
In this thread someone pointed me to use sqlalchemy-migrate to help with a fast-changing web application using sqlalchemy.
However a Do It Yourself method was also recommended consisting in manually writing CSV columns for the new database schema, and finally import them.
The problem is that I can't find real-world examples of sqlalchemy-migrate. Ressources that I have found at best decribe adding a single column or a column rename. The official documentation essentially describes the API and it's hard to see how to use migrate effectively. From the doc I cannot even know if migrate could help changing the database engine, from sqlite to mysql for example, while the DIY solution would to the job.
I really want to see code that would make some non-trivial transformations of a database schema and proving that migrate is really a useful tool.
Where can I find good examples/tutorials for sqlalchemy-migrate ?
Thanks !
Don't forget about google code search when looking for real work examples of code. For instance the follow search:
http://www.google.com/codesearch?hl=en&lr=&q=%22from+migrate+import%22+lang:python&sbtn=Search
Will pull up a number of real migration scripts. It basically looks for Python files with "from migrate import" in the file.
Work through some of these and see if you can follow what they're doing.
I have a shapefile that I would like to upload to a spatial SQL Server 2008. I have tried using this tool: SQL Server 2008 Spatial Tools. But without luck.
Does anyone know any other (free) tools for doing this?
You can use ogr2ogr to convert from shapefile to GML (or many other formats) and then use SQL Server's GeomFromGML to import. You will need to call GeomFromGML for each feature in your shapefile, but that's a relatively easy program to write.
I have written code in arcbjects for do this task. If you have license arcview or engine you can create a console application in c# and use this code: see https://gis.stackexchange.com/questions/33917/how-to-import-shapefiles-into-ms-sql-2008-and-then-view-that-data-using-qgis?lq=1
UPDATE: I decided I would just point people to the official Github repo instead: https://github.com/zer0infinity/OGR2GUI
This [ogr2ogr fork] tool will attempt to parse the content of the input file (in my case, it was a shapefile) and output it as a bunch of different formats (in my case, I needed it as a CSV, but you can even export your file as a SQLite file). Unfortunately it doens't do straight up SQL, but you can do a dump from a SQLite viewer such as SQLite Browser and import it onto MS SQL. I did notice some inefficiencies when converting to SQLite (I lost some attributes/tables). I also tried MobileMapper Office (MMO) (with better luck) to export to CSV and it did preserve a lot of the data, but you'd then need to write a script to import the data to SQL. If you are going that route, let me know, I'm currently writing a VBA script to deal with the exported data from MMO.
If you're hoping to build this into your application (a script of some sort), you may have some luck trying with ogr2ogr, but you never know what the data is going to look like. documentation is found here http://www.gdal.org/ogr2ogr.html
Original answer: Save yourself sometime and just use this amazing tool: http://ogr2gui.ca/
It's based off ogr2ogr, but with a nice GUI.
I've made an app for importing shapefiles into SQL Server. It's made primarily to suit my needs but i had some spare time so i made an installer for you.
Some nice things you can do with it:
choose encoding of input shapefile
rename/remap destination table column names
choose the destination table name
set the primary key for the destination table
It has a user interface and you can download it for free.
More details can be found on my blog, here's the link: Import shapefiles into SQL Server
I often have data in Excel or text that I need to get into SqlServer. I can use ODBC to query the Excel file and I can parse the text file. What I want though is some tool that will just grab the data and put it into tables with little / no effort. Does anyone know of such a tool?
Have you tried the SQL Server Import/Export Wizard ?
In SQL Server Management Studio, right-click your Database Name, and select Tasks menu, Import Data. For Data Source, select Microsoft Excel, browse to the .XLS...
If you are using Sql Server look at Integration Services (SSIS).
You can also take a look at parse-o-matic
Use DTS or SSIS depending on which version of SQL Server you have. There is an import wizard which can get you started, but data imports are rarely simple and usually involve some sort of data cleanup so that your incoming data is acceptable to the table where you intend to store it. Excel data, in my experience, is usually particularly bad inthis respect becasue it often isn't stored properly in Excel to begin with.
I haven't seen commercial tools that do this. I create this kind of tools at work all the time, and the data validation is not trivial. This just makes sure that you don't have bad data making it into your database.
I found that for simple data conversion needs something like FileHelpers is pretty good. It still needs programming though. This framework is fairly easy to use, and somebody with a little bit of experience could bang something out for you.
On further thought, you can use the SQL Server bcp utility to upload the contents of a text file. This is a command-line utility and has a lot of switches. I would suggest you experiment on a test table before you use this in a production table.
It's been a while since I used it, so I can't remember if you can directly use an Excel spreadsheet. Text files are always the easiest to deal with in any case.
Seems like it'd be pretty easy to write a script that reads the text file, and converts it to "INSERT * into TABLE" Sql statements. I suspect this has already been done, but a simple implementation would be less than 100 lines of code in your favorite scripting language.
Hey, Google says SQLServer comes with such a tool, BULK INSERT: