I am currently looking for ways to convert data from JSON format to Modbus format, ideally a JS script that is readily usable in Node-RED. I thought such wouldn't take long to search given the wide application and usage of the two formats (especially in the IoT space) but boy I was dead wrong. After scouting thoroughly on YouTube and Google for a week, I failed to find any useful resources. However, when I inquired a few field veterans on this, they all told me it's a simple to write such conversion code and wanted to charge me for a day rate.
Question: Does anyone here knows how to do it/generous enough to share a sample JS code of such conversion? I believe script customisation will be heavily needed but having a sample to based on is really all I need now. Thanks!
Related
TL;DR: I'm looking for some resources on generating GRIB2 data sets on the fly, ideally using in-house-generated wind data in a CSV format.
We have a bunch of data for a series of localized weather stations monitoring wind information around our city. They report in at ~2-3 minute intervals (far more frequent than standard weather data), and from their reports we have lat, lon, wind speed, and wind direction. Someone went and told the boss about these really slick visualizations, like this that can display wind speed and direction, and it's my job to make it happen.
The above plug-in for Leaflet, GitHub here, as well as several others, all use GRIB2 data, which from my research involves a left/right set of data and an up/down set of data for a series of points plotted out across a region.
The problem I'm having is that I've only found a handful of tools that interact with GRIB2 data, and most seem to decode data from the GRIB2 dataset, and only one tool running on Fortran seems to exist that compiles GRIB2 data together.
So, is there any way to generate GRIB2 data on the fly using proprietary data at 2-3 minute intervals?
I've gone through this resource on NOAA's website, which is where I found a few tools.
I know how frustrating it can be to work with GRIB and some of the other science/weather related formats. This may not be the best answer, but it might be your only answer as I find these types of questions to only gather dust because of the general lack of knowledge with the formats and tools.
From what I remember, CDO tools (link here) can do some magical things - but I am not that experienced with it. I do use it for converting satellite data to plain text and it's been an absolute lifesaver! So I will explain :
My suggestion was to first convert the CSV to netCDF. I had a link saved for this a long time ago, but never came to really needing it. (discussion here). Essentially, some python code should be able to do the conversion for you. There may be several ways to do this, but I have never looked into it beyond initial research.
Next, you should be able to convert .nc to .grib using CDO. I know it can do quite alot. Here is a discussion regarding this, so it must be able to be done.
I also see at this link where someone converts grib to netcdf, but you should be able to do it in reverse as well. I just don't know the exact commands. From the link :
As an example of use of CDO, converting from GRIB to netCDF can be as simple as
cdo -f nc copy file.grb file.nc
I would suspect its just the reverse but probably something like :
cdo -f grb file.nc file.grb
Hopefully you can put things together for it to work without being too hack-y.
You can do this in a simple python script using pandas , xarray and cfgrib
import pandas as pd
import cfgrib
data = pd.read_csv('your_csv_data.csv')
xarray_data = data.to_xarray()
cfgrib.to_grib(xarray_data, 'out2.grib')
Please note that you have to define grib specifications first before you store as grib data.
I tagged this question as "conceptual" because I'm not sure if creating Chrome Extension is the best way. In my opinion it's better to ask before spending few hours writing something and find out that some part is too much difficult or impossible.
The problem
I used to analyze my finance using csv files downloaded from bank account. But as it sometimes happens, my bank launched new GUI and csv feature disappeared. They do not know when and whether they will do it at all. So I have to grab transactions in some way and put them in CSV file.
Concept of solution
I think scrapping the page with transactions it's not good way, because whole data is looking like generated with totally random CSS classes and ids. I noticed that list of transactions is sent in JSON format by AJAX response. I analyzed that JSON and every interested field has name, so access to data is quite easy. Only one problem I see, it's that first JSON load shows only first 10 transactions. To load more I have to scroll down, then there is next AJAX request to the same URI, then next JSON load comes in response. So, if I want to get transactions from whole month I have to scroll few times down and my tool should catch first and also next responses.
I don't have experience with Chrome Extensions but they claim that if I know web technologies like JS, CSS, HTML it shouldn't be difficult to write simple extension. If I can take this JSON from AJAX response to my extension then generating CSV file shouldn't be problem.
The question
The main question is if my concept is possible to realize. Is there easy access to data loaded from AJAX response? If you see any better solution I'm open to suggestions.
you can create CSV file from Json in DOM (don't required to call any API).
Please refer this demo link
jsfiddle.net/hybrid13i/JXrwM/
if you care implemented this functionality in chrome extinction so you need to add download permission in manifist.json https://developer.chrome.com/extensions/downloads
As part of my Master's thesis, I'm trying to run some statistics on which factors affect whether crowdfunding campaigns get funded or not. I've been trying to get data from the largest platform Kickstarter.com. Unfortunately, they have removed all the non-successful campaigns from their website (unless you have the direct link).
Luckily, I'm not the only one looking for this data.
Webrobots.io have a scraper robot which crawls all Kickstarter projects and collects data in JSON format (http://webrobots.io/kickstarter-datasets/).
The latest dataset can be found on:
http://webrobots.io/wp-content/uploads/2015/10/Kickstarter_2015-10-22.json_.zip
However, my programming skills are limited, and I don't know how to convert it into an excel file where I can manipulate the data and run my analysis. I found a few online converters, but the file is far too big for it (approx 300 mb).
Can someone please help me get the file converted?
It will earn you an acknowledgement in my Master's thesis when it gets published :)
Thanks in advance!!!
I guess the answer for this varies massively on a few things.
What subject is the masters covering? (mainly to appease many people who will probably assume you're hoping for people to do your homework for you! This might explain why the thread has been down-voted already)
You mention your programming skills are limited... What programming skills do you have? What language would you be using to achieve this goal? Bear in mind that even with a fully coded solution, if it's not in the language you know, you might not be able to compile it!
What kind of information do you want from the JSON file?
With regards to question 3, I've looked in the JSON file and it contains hierarchical data which is pretty difficult to replicate in a flat file i.e. an Excel or CSV file (I should know, we had to do this a lot in a previous job of mine).
But, I would look at the following plan of action to achieve what you're after:
Use a JSON parser to serialize the data into a class structure (Visual Studio can create the classes for you... See this S/O thread - How to show the "paste Json class" in visual studio 2012 when clicking on Paste Special?)
Once you've got the objects in memory, you can then step through them one by one and pick out the data you want and append them to a comma-separated string (in C# I'd use the StringBuilder) and write the rows of data out to a file on disk.
Once this is complete, you'll have the data you want.
Depending on what data you want from the JSON file, step 2 could be the most difficult part as you'd need to step into the different levels of the data hierarchy.
Hope this points you in the right direction?
You may want to look at this Blog.
http://jdunkerley.co.uk/2015/09/04/downloading-and-parsing-met-office-historic-station-data-with-alteryx/
He uses a process with Alteryx that may line up with what you are trying to do. I am looking to do something similar, but haven't tried it yet. I'll update this answer if I get it to work.
I have recently been tasked with converting several documents from EZ-Letter format to HTML, or at least to RTF, from which I can convert. I do not have access to the EZ-Letter application and am having a tough time finding a solution online both due to the keywords and the likely small-sized community. I'll be asking the company directly during their office hours later this week to see if a solution exists, but asking on SO never hurts.
The documents I intended to convert to HTML were simple enough that I coded the HTML by hand using document images processed through EZ-Letter as reference.
Horribly worded question...I know.
I'm working on an application that processes data for the previous day. The problem is that I know the customer is going to eventually ask to it for every hour or some other arbitrary time interval. I know that languages such as Java or SQL have masks for defining dates. Well what about a way to define a time interval?
Let me ask it this way. If someone asked you to create a configurable piece of software how would you allow the user to specify the time intervals?
UPDATE
What I'm looking for is a textual representation. Imagine somethign that would be put into a config file.
Here's how Google App Engine's cron API is configured, it's a bit more user-friendly than cron:
http://code.google.com/appengine/docs/python/config/cron.html#The_Schedule_Format
Python (and Java?) source to implement it should be available in the SDK. Haven't looked to see how easy it would be to extract, but it should at least provide some ideas.
I think it'd be possible to add various things to the format as required - it's structured enough to be extensible. For example it's currently missing the ability to say "every hour at 4 minutes past", which is common in UNIX cron but not really relevant to GAE because although perhaps the engine could figure out which minutes are busy and which aren't, and balance load, the user certainly can't.
Obviously the major weakness of offering something that looks like natural language to the average user, is that they'll think your code is psychic, and expect it to also understand things like "every other Wednesday except the week after Easter", or "whenever the clocks change" ;-)
I'm not sure if I understand your question correctly. Are you asking for a way to input time intervals?
I don't know your environment, but most language frameworks offer some more or less sophisticated GUI components. Something like a calendar control may be what you are looking for?
Edit: Ah, command line (or config file) it is.
Can you parse an XML file? In this case, you could just serialize a timespan (most languages have a class like this). The use may edit the XML file, typically just replacing, say, the value of the <minutes> part. Deserializing this XML file to obtain a Timespan again is easy.
If you prefer plaintext or command line input, it is a bit less easy. However, many languages support some kind of Timespan.Parse (string text, string format). You should have a look if this concept is present in your environment.
Many environment offer some kind of sscanf(), that parses an input string. Sometimes there is a "time" format as well.
If the earlier ideas don't provide a solution, you can try to parse the date using regex. There should be many resources on this topic on this site as well as the web.
If you dislike regex, split the input string according to your own format rules. That's kind of an ugly solution though.
If you don't feel like splitting user input stringy by hand, use Random.NextInt(1000) and hope nobody notices. 0:-)