I start to use dygraph and I found some difficulties to work with independant series, specially because I have a voluminous data, is there any solution to work with independent series without formating data to especial format recommended by dygraph?
Thank you for your help
You can use a Google DataTable to manage large data series.
Related
I have a big shape file (about 1GB) and I need to display and update data from this file with
an online map (mapbox).
As I understand I need to convert this shapefile into some DB (Spatialite? PostGIS)
and then query it by map BBOX to display by parts.
What is the easiest way to do this? Thanks
Thats a huge topic and depends very much on the usecase. If you want to dive deeper into the topic I would recommend this articel. It describes the underlaying theory and gives a working example for a SQL-Query. For me, it was sufficient to get going with a viable prototype. I think Postgres (with it's Postgis extension) would be a good choice since the data can be directly served as mvt.
I found this site very useful for displaying GeoJson on the fly, it usually copes well with large data, failing that you can import it too. https://geojson.io
Even though I am quite familiar with the concepts of Machine Learning & Deep Learning, I never needed to create my own dataset before.
Now, for my thesis, I have to create my own dataset with images of an object that there are no datasets available on the internet(just assume that this is ground-truth).
I have limited computational power so I want to use YOLO, SSD or efficientdet.
Do I need to go over every single image I have in my dataset by my human eyes and create bounding box center coordinates and dimensions to log them with their labels?
Thanks
Yes, you will need to do that.
At the same time, though the task is niche, you could benefit from the concept of transfer learning. That is, you can use a pre-trained backbone in order to help your model to learn faster/achieve better results/need fewer annotations example, but you will still need to annotate the new dataset on your own.
You can use software such as LabelBox, as a starting point, it is very good since it allows you to output the format in Pascal(VOC) format, YOLO and COCO format, so it is a matter of choice/what is more suitable for you.
TL;DR: I'm looking for some resources on generating GRIB2 data sets on the fly, ideally using in-house-generated wind data in a CSV format.
We have a bunch of data for a series of localized weather stations monitoring wind information around our city. They report in at ~2-3 minute intervals (far more frequent than standard weather data), and from their reports we have lat, lon, wind speed, and wind direction. Someone went and told the boss about these really slick visualizations, like this that can display wind speed and direction, and it's my job to make it happen.
The above plug-in for Leaflet, GitHub here, as well as several others, all use GRIB2 data, which from my research involves a left/right set of data and an up/down set of data for a series of points plotted out across a region.
The problem I'm having is that I've only found a handful of tools that interact with GRIB2 data, and most seem to decode data from the GRIB2 dataset, and only one tool running on Fortran seems to exist that compiles GRIB2 data together.
So, is there any way to generate GRIB2 data on the fly using proprietary data at 2-3 minute intervals?
I've gone through this resource on NOAA's website, which is where I found a few tools.
I know how frustrating it can be to work with GRIB and some of the other science/weather related formats. This may not be the best answer, but it might be your only answer as I find these types of questions to only gather dust because of the general lack of knowledge with the formats and tools.
From what I remember, CDO tools (link here) can do some magical things - but I am not that experienced with it. I do use it for converting satellite data to plain text and it's been an absolute lifesaver! So I will explain :
My suggestion was to first convert the CSV to netCDF. I had a link saved for this a long time ago, but never came to really needing it. (discussion here). Essentially, some python code should be able to do the conversion for you. There may be several ways to do this, but I have never looked into it beyond initial research.
Next, you should be able to convert .nc to .grib using CDO. I know it can do quite alot. Here is a discussion regarding this, so it must be able to be done.
I also see at this link where someone converts grib to netcdf, but you should be able to do it in reverse as well. I just don't know the exact commands. From the link :
As an example of use of CDO, converting from GRIB to netCDF can be as simple as
cdo -f nc copy file.grb file.nc
I would suspect its just the reverse but probably something like :
cdo -f grb file.nc file.grb
Hopefully you can put things together for it to work without being too hack-y.
You can do this in a simple python script using pandas , xarray and cfgrib
import pandas as pd
import cfgrib
data = pd.read_csv('your_csv_data.csv')
xarray_data = data.to_xarray()
cfgrib.to_grib(xarray_data, 'out2.grib')
Please note that you have to define grib specifications first before you store as grib data.
It seems most Bokeh embedding examples are using bokeh.plotting.figure object. Is it possible to embed a high level chart, like bokeh.charts.Bar or bokeh.charts.Scatter? Or is it possible to have convert a high level chart to a bokeh.plotting.figure object?
Thanks a lot.
The User's Guide section on Bokeh APIs has a good run down of how all these parts fit together, that I would suggest reading.
The long-story-short: Regardless of what API you use, bokeh.plotting or bokeh.charts, the end result is always just a collection of the same low-level bokeh.models objects. You can can think of bokeh.models as very basic building blocks, and the other higher level APIs as conveniences that help you to assemble the building blocks more efficiently and correctly.
So, in that light, yes, it is perfectly fine to embed a bokeh.chart using exactly the same functions described in Embedding Plots and Apps.
The one thing I will add is that if you need to update the plot's data after the fact, in place, then the bokeh.figure API will probably be more straightforward. The mapping between your data, and what gets plotted is more direct. Things generated by bokeh.charts may transform your input data into entirely different forms before plotting (e.g. you give a series, and Histogram has to spit out coordinates for boxes—not the data you started with)
I have a fairly simple mysql database. I want to extract data from it and display as a graph to analyze it. Nothing advanced, just line diagrams and such. As a programmer I can always write some code for this myself. But does anybody know of a program that can load data (using sql queries?) and display it in various graphs?
Checkout DBPlot for Windows environment.
You can also have a look at Google Charts, if you want your graphs displayed in web pages.
Don't know what type of output you're looking for, but FusionCharts has helped me a lot... http://www.fusioncharts.com/free/
You can use GNUPLOT if you are in linux environment. I am currently using this for generating automated EOD reports which contains graphs. The graphs are drawn using the data fetched from mysql (Currently we are using in csv) But gnuplot is not as good as Fusion Chart in UI Perspective
I'd have to recommend JQplot if you want these graphs in web pages - it's non Flash (hooray!), uses JQUery (hooray!) and has some very, very cool features (hooray! - check out the examples page).
As per the docs page, you can just pass an array of values to it and it'll plot what you want, and generating an array is simple in server side languages such as PHP.
Hope this helps,
James
An extremely flexible library for graphs (images and applets) would be JFreeChart.
On the other hand, if what you actually need it is a reporting engine, Jasperreports should be your first and final stop but you will have to do A LOT of reading before starting doing something with the latter and it won't be time lost for nothing.
Try Smart chart maker it is used to create many type of charts that can be fed from a MySQL database . supported chart types include line charts , pie charts, bar charts, and scattered charts.