I'm using folium.GeoJson to plot polygons on a map. I have one dataset with different fields/attributes; I'm using the same data for each folium.GeoJson layer but with various fields to display different attributes and add them in layer control. I guess this makes the final Html file big in size.
One solution I found to reduce the Html file is to minify it, but the slimmer package isn't available in anaconda.
Do you have any suggestions for this case?
You can reduce the size of the HTML file dramatically by using mapshaper to reduce the size of the geojson file.
Drop your file or search and import. I check the box "detect line intersections" and upload:
Choose your method of simplifying the coordinates (I use Douglas-Peucker method, and prevent shape removal):
Then set the simplification (you can play with the settings to get the desired result, I found 3% worked well)
Then export the file (it exports as a .json file so remember to change to .geojson:
This reduced the file from 17.6MB to 614KB and the outputted HTML file from 222MB to 7.58MB
Found a way: to simplify spatial data that significantly reduces data size.
In my case, I was using shapefile as the spatial data, particularly polygons, that is, the collection of points (vertices) and highly affected the data size. I just used the simplification tool that removes extraneous vertices but preserves essential shape. I used the Simplify Polygon tool in ArcMap, but the same is possible in python with the geopandas.GeoSeries.simplify function and in R with the 'sf' library.
Related
I want to display the kml file in cesium globe, and as described in the cesium's workshop code, we need to load the file by passing it with the file location (or URL). The following line of code is the specific line of code that cesium loads the kml.
var geocachePromise = Cesium.KmlDataSource.load('./Source/SampleData/sampleGeocacheLocations.kml', kmlOptions);
The entire code is available here
My question is: I have the longitude, and latitude, (and height) information saved in variables and instead of always saving them into .kml file and then load them via folder, I want to pass this information to cesium kmlDataSource (the code above) directly.
It would be great if anyone has any solution to this.
Please let me know if further information or code snapshot is required. Thanks
If you already have the information you need stored in JavaScript variables, there's no need to export to KML and import it back into Cesium. You can directly add the indicators you need as Cesium Entities, which is what the KML loader is creating when it reads a KML.
Typically, a KML-like pin is represented by a Cesium Entity containing either a point or a billboard, and optionally an associated label.
Here are some relevant demos that show how this is done:
Billboard demo
Map Pin demo
Label demo
Each of these demos calls viewer.entities.add({ ... }) along with a position for the Entity and some sort of graphical indication(s) to display to the user. You may place one of each on an Entity, for example a billboard and a label are often both defined when adding a typical KML-like Entity.
If your data is stored on the server however, you will need some mechanism to stream it to the browser. CZML is Cesium's native format for doing so, but KML is also available as an alternative for certain kinds of graphics. You may also use any API of your own design, and create Entities when the data becomes available in JavaScript.
How can I edit a large JSON manually?
I have a large JSON file, about 100 MB. I'd like to manually inspect some attributes, and then add more attributes to some of the objects.
I'd start off by looking at a subset of the file. Say, the 1st 100 objects. I'd gradually scale up to looking then at maybe 250, then a thousand, etc.
Can someone suggest a language or software (I'm running Windows) that excels at this task?
Some previous suggestion that aren't working or can't work.
Sublime - Could never load the file. Loading bar forever. Had to kill.
NotePad++ - Could never load. Froze. Had to kill.
Anything online - The data is confidential.
More Python and Jupyter information.
with open(path, 'r') as f:
data = json.load(f)
for i, (k, v) in enumerate(data.items()):
print(i, k, v)
if i == 2:
break
Causes an error. I think it has to do with Jupyter, but I'm not sure.
IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_data_rate_limit`.
Current values:
NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
NotebookApp.rate_limit_window=3.0 (secs)
That makes me wonder if going about it this way is just dumb.
Possible Solutions
Build a custom app using TKinter
Just don't use a Jupyter Notebook
What you can do is to write a simple GUI program. use TKinter, to create a window and a text area inside it to show the json, a text box where you will input, how many objects you want to see, and a button named Next or something to see next and one more button to save.The following will be the functionalities for each of the items.
First you will be reading complete json in python and making it a dict.
Next Button - This will keep iterating based on the value in the TextBox. you could write a custom generator, where it will be yielding based on the number of values required.
Save Button-: This will keep saving the current json into a new json or if you could, you can try to write a function to update the current json directly.
Text Area - you should take the dictionary and convert to json and show the output from the Next Button's generator.
If you are using linux (or have an opportunity to transfer the file to *nix) you might wish to check out for number of lines within a file via
wc -l myfile.json
Let's say, for the purpose of simplicity, that your file has 2530000 lines and you wish to split it into 100k lines each, you can utilize any of the commands available at your distro to split the file further into desired chunks and then to edit them, one by one.
If you are comfortable with going the "linux way", check out some of the hints given on other topics, i.e.
edit multi-GB file when vi editor doesn't work
I hope it helps!
The only viewer I have used that works on large files (I had up to 250MB size files) is Dadroit. It is fast to view and comes with search.
Now, to edit, I use vi. I search for the location and make local edits. Vim or another simpler editor should work on Windows. Have you tried vscode? 100MB shouldn't be too large for it.
The other awesome terminal tool for viewing and editing data is Visidata. I have had mixed luck with it working on json files.
Not the best answer, but the problem with reading the JSON seems limited to Jupyter Notebooks (or even the limitations of my laptop).
Working in Spyder or running from the command line circumvents the Jupyter error mentioned in the original question.
It'd be great if someone knew how to tweak Jupyter to avoid this problem (sorry, I'm not sure how yet).
for editor,try notepad++
for language, try Python
since you haven't give your data structure, I can't give more answer.
Since Dygraphs apparently does not have any functionality for adding separate series of data to a chart one at a time (but rather only loading all the data series of a chart at once from a CSV file, or an in-memory array of arrays) I'm looking to make some code to do this myself.
My reason? My problem/scenario is that I have a "base file" containing a series of data many million values large. I will then need to show many separate charts that display this large data series TOGETHER with a bunch of other respective smaller data series, and I'd very much rather not duplicate the large dataseries in a new CSV file on disk for each such chart, but rather first load the big "base data series" from the CSV "base file" directly from my Javascript, and then for each such chart integrate one such smaller data series with it before sending it off to rendering by means of a new Dygraph(...) call.
The CSV file loading functionality that already obviously exists somewhere inside the Dygraphs code is very nice, so I'd very much like to use it for this loading of the large "base data series" if possible, from a single separate CSV file.
So, in short, the question is:
How can I use the existing CSV file loading functionality of Dygraphs separately from inside my own code, in order to load arbitrary CSV files into the Dygraphs chart data array format in-memory, so that I can finally merge these multiple data series arrays using my own custom code?
What I'm hoping for is something like this:
loaded_data_series_1 = some_secret_internal_function_or_method_of_dygraphs('file1.csv');
loaded_data_series_2 = some_secret_internal_function_or_method_of_dygraphs('file2.csv');
merged_data_series = my_own_custom_dataseries_merging_code(loaded_data_series_1, loaded_data_series_2);
g = new Dygraph(document.getElementById('my_chart'), merged_data_series,{});
The key here would thus be to know what some_secret_internal_function_or_method_of_dygraphs() should be replaced with for this to work.
Could the Dygraph devs or anyone else possibly point me in the right direction here?
(I tried to look inside the Dygraphs code myself, but unfortunately got lost pretty quickly due to insufficient Javascript coding skills on my side)
I am trying to export all the plots of my NetLogo model after simulation runs in a csv format with the primitive export-all-plots.
I haven't found yet the way to open this csv file with an external reader in order to get more clear plots. I tried with gnuplot but it looks like it's not able to open the csv format created with NetLogo:
"export-plots data (NetLogo 5.0.5)"
^
"C:\results\interface.csv", line 1: invalid command
How can I open csv plots with an external reader?
There are two complicating factors about NetLogo's plot export format. First, there's a three line header at the beginning (plus an empty line after) that just gives information about the model and when the data was generated. Next, there's data about the model settings, the plot state (pen colors and such). Finally, there's the data itself, which itself is somewhat complicated by the fact that you can have multiple pens per plot. So I'm not surprised gnuplot couldn't read it as is.
The table's are quite easy to use in GUI spreadsheet application, like Excel, LibreOffice's Calc, or Gnumeric. You can just select the data you want and generate the plots.
To do this at the command line, I'm afraid you might have to write a script to read it in. This should be pretty easy in something like Python or R. Just skip the metadata lines, and use a CSV parser to read in the rest.
You might also try using BehaviorSpace to generate the data, but make sure to use the table output. It let's you generate the data from many runs at once, and the format is a little more consistent. There are still 6 lines of metadata at the top, but you can just delete that. I believe this is more the standard practice in NetLogo.
I have a problem with JSON format.
I am going to generate a 2D building plan in JSON format automatically. Then this JSON file is loaded to Three.JS for visualization.
Currently, I can easily create various geometry type in JSON format based on:
https://github.com/mrdoob/three.js/wiki/JSON-Model-format-3.1
However, I need to know more about this data format.
Firstly, how can I add text to the model?
Secondly, I could not find line as a geometry in this format. Is there?
Thirdly, is there any document that I can refer to it for more explanation regarding this
format?
Lastly, how can I add additional information about geometries
inside the JSON format? for example when I create a polygon in JSON
format, I want to add additional information regarding that polygon
such as area, owner,...
Either through a texture, or by creating THREE.TextGeometry and merging that with the other geometry
AFAIK, no. You might be able to emulate lines by loading a separate object that you render as wireframe.
I've used the source code: Loader, JSONLoader
I don't think that's possible while maintaining compatibility, i.e. you would need to patch Three.js loader.
One option for achieving what you want could be to add custom entries to the JSON, e.g. "lines": [], "customFaceProperties": [] and then parse them yourself once the stock Loader has parsed everything it understands.
Also, since it seems you might be creating something that has different things in it, you might want to take a peek at the scene loading capabilities: example, SceneLoader.js