I need to create a daily process which pulls market data from a website http://www.apxendex.com/index.php?id=137.
I had originally created an excel+VBA sheet which added the data to an xml file and this worked fine. However, the machine which I will be putting the code on doesn't have excel so all my work was pointless (stupid I know).
I'm not looking for someone to write anything for me, but some general tips on where to start, what language to go for etc. would be helpful.
Thanks
How did you get your information from the website?
There is a DataCapture API for that site. You can find the manual on this site (you have to login to see the manual). I guess it would be the easiest way to go the recommended way.
Related
I have to do the following steps two or three times a day
log in into Elasticsearch
Go to Dev Tools
Run a specific query by selecting it and pressing ctrl + enter
Query that I have to run
Select the results that returns in the "buckets" and copy it.
The yellow markdown in the image is what I have to select and copy
Then I go to https://www.convertcsv.com/json-to-csv.htm and paste the results so it converts to CSV.
Where I have to paste the results.
I can then download the CSV and then import it into google sheets so I can view the results in a Looker Dashboard.
Button to download the converted CSV.
This take me some time everyday and I would like if there is any way that I could automate such routine.
Maybe some ETL tool that can perform at least part of the process or may some more specific way to do it with python.
Thanks in advance.
I don't have much experience with what I want to do and I tried to search online similar issues, but couldn't really find anything useful.
I don't know you tried, but there is a reporting tool on elasticsearch inside the "Stack Management > Reporting". On the other side, there are another tools which you can work from a server with crontab. Here are some of them :
A little bit old but I think it can work for you. ES2CSV. YOu can check there are examples inside the docs folder. YOu can send queries via file and report to CSV.
Another option which is my preference too. You can use pandas library of python. YOu can write a script according to this article, and you can get a csv export CSV. The article I mentioned is really explained in a great way.
Another alternative a library written in Java. But the documentation is a little bit weak.
Another alternative for python library can be elasticsearch-tocsv. This one is a little bit recently updated when I compare it to first alternative. But the query samples are a little bit weak. But there is a detailed article. You can check it.
You can use elasticdump, which is written on NodeJS and a great tool to report data from elasticsearch. And there is a CSV export option. You can see examples on GitHub page.
I will try to find more and I will update this answer time by time. Thanks!
I'm trying to make a website for an assignment. In this assignment, I have to include examples of some of my work and I want to include a database I made in Microsoft Access, is there anyway I could do this? It preferably has to be in html. The database is just a table, a form and a report.
HTML can't to connect to database alone, you should to use other tecnology together to this be possible, a better way to do it in a web system is to connect through a API, if not know about, google it, have a lot of documentation and tutorial to make modern restful API, for your case that is using microsoft access I found this Microsoft tutorial maybe can help you.
-> https://support.microsoft.com/en-us/help/308459/how-to-connect-an-html-page-to-a-microsoft-access-database
Basically I have a web app which it contains items and the people can comment on it and tell what they think about it. just like youtube videos or any other similar website.
Question
Where is the best place to save these comments. Do I Create a table in MySql database and save it there or do I save it in a .txt file and then save the location of the file on the database. I would really appreciate if someone tell me which is good from performance perspective or is there any alternative better way.
Much appreciations.
Save the comments directly in database.
Adding a comment in file and saving the path in database is not good you are adding extra work.
Reading from database is easy and more professional as every where it is in use. In YouTube as well.
There is chance someone delete your file and you will lose your data.
Clearly the database. They were build to replace the unpleasant storage of data in flatfiles also for performance reason.
You need to save the comments in the database. It is easier to query and sort the comment later on. Despite save as .txt, it is safer to save it in database.
What I have been trying to do for some time is to create a basic (online) text editor that saves files on to the web server.
So far I have had no luck mostly because a lot of examples seem to use a PHP uploader script to save the file and the web hosting service I use seems to have their PHP set up to not allow uploads and there is no way I can change this.
So after no luck elsewhere I decided to see what suggestions you might have. Any suggestions as to what to do?
Find another host. Are you using a free host by any chance? Considering how cheap they are these days (I pay £30ish per year, which is totally worth it as a playground for me), you should invest!
I am building a website with ROR 3. I need to provide a page to my clients wherein he could edit his pricing info regarding the application. I am quite confused on how to do this. The pricing page needs to be displayed as an html table with different columns which has got the pricing info.
I am thinking of different ways to do this.
1) Allow the client to create and upload an html page and then save it as a file in my public directory and render as an when the client clicks on the pricing link.
2) The clients may not have bare technical knowledge, hence make the client upload some other formats like Word, Excel etc and then parse it and store it as an HTML file in the public directory.
3) Provide the client with some real time editing tools where in the client could edit in a fixed format, and after wards save the file and render it later.
Also, I wouldn't like to store these infos in my database. There would be quite a few number of clients and hence managing all these data in my database would become cumbersome. Storing all these as plain html files and rendering it later would be the most ideal thing for me.
There might be other better steps in doing this as well. Could you please suggest which might the better, or any other option that could suit my needs? Basically I would want my clients to have a mechanism where they could provide there pricing details, edit it later and display it back as an html table, all this without using an Database backend. Any suggestions would be mostly appreciated.
Good way is Excel(csv format).
You can do PHP with Excel. I thing this is the best solutions for your requirement.
Try this.
http://php.net/manual/en/function.fgetcsv.php
If you are give authority to user to change edit contain and you have to used " CSV or Excel" please see these links:
Importing CSV and Excel
Exporting CSV and Excel
If you really don't want to use database then you can use YAML as a structured storage.
e.g. ( you, most probably, could come up with a better structure )
SMS_Pack:
Sl_No:
1: 10000
2: 25000
3: 50000
You can read those .yml files and parse them as hashes. Should be fairly easy to represent that hash as a HTML table.
For the creation, I'm sure you can come up with some dynamic form input. Or to just let the client send this kind of file ( which might not be the best solution ).
But it just might be easier to manage all of this information within a database.