I'd like to access parquet files on HDFS and would like to use arrow library. However, I need to know what other dependencies need to be installed on the client side? For instance, the following page explained what dependencies are required to use arrow for python (pyarrow) but I can't find those dependencies for R.
As of 0.14.1, the R arrow bindings don't yet have HDFS support. Your best bet at the moment is probably to sync the files to your local filesystem before working with them.
Related
I want to do some spatiotemporal data analysis with kepler.
As the data is confidential and also huge, i cannot upload it in the demo site.
Tried installing keplergl locally but it needs a Mapbox Access Token (which I think is paid).
Is there a way I can use kepler with OpenStreetMap. (I want to run it in jupyter notebook or through python OR a one-time react setup, since I am not familiar with react).
Also when I use Kepler in jupyter notebook empty map loads as shown below:
Without data map
as soon as I load data it goes away:
With data map
here is console output:
error
My jupyter notebook is also configured with
jupyter nbextension install --py --sys-prefix keplergl
jupyter nbextension enable keplergl --py --sys-prefix
Is there a way to fix the jupyter notebook error OR is there an alternative open-source tool like keplergl.
Saw some solutions using Tile with docker but was not completely clear. I would prefer a way in which I can use it in jupyter notebook.
Tried pydeck of DeckGL, wasn't very interactive.
For example, I have multiple columns in my database and kepler lets me filter based on each column.
Also, kepler allows one to select which different columns for weights.
If pydeck allows it can you please say how?
This question comes from my poor understanding of how GDAL/OGR is supposed to work server-side, but before trying it I'd like to know if what I want is feasible with this library.
Basically, I am developing a webGIS application where the user should interact with a WFS. I'd like to give the user the possibility to download the data from this WFS as a format that the server does not provide as output so I need to convert it first.
I need the function ogr2ogr with the WFS driver to be able to convert the GML response of the server in (e.g.) Shapefile format.
My very basic question is: how can I call ogr2ogr which is installed on my server when the user click on (e.g.) a "Download" button?
My idea obvioulsy is to tie the onclick event with a JS function that calls ogr2ogr, but here my lack of knowledge in server-side web development does not le me go on.
Am I supposed to prepare a script (like a PHP) that internally calls ogr2ogr, or should I place the GDAL/OGR executables in a cgi-bin folder so that I can point and calling them with the required parameters?
I'd suggest investigating your server-middleware of choice; there's an excellent chance it supports OGR2OGR already if it's Open Source. The big two:
GeoServer has an ogr extension that can do what you want - https://docs.geoserver.org/stable/en/user/extensions/ogr.html
MapServer supports OGR too - https://mapserver.gis.umn.edu/output/ogr_output.html
Alternatively, you may want to consider investigating the WPS standard - but it's probably overkill if you're just changing formats.
This is a very basic question, but I have not been able to find the answer anywhere.
I just got Cocos 2Dx 3.3, made a new project and built the HelloWorld scene that was generated.
It generated projects for all platforms, but I am currently compiling and running the Mac project using XCode.
I can easily add new files using XCode, but of course it only adds it to my XCode Project. I would have expected a way to automatically modify all projects at once to add files or change compiler settings. I saw that there is a CMake file, probably used for command line compilation, but I cannot find a way to use that to regenerate the projects for all platforms.
Is it possible to automatically add a file to all platforms? Maybe it is possible to modify the template directory os Cocos and use the cocos new command line to recreate the project from scratch?
Or is it better to do all that manually?
Such an automation would imply parsing of a project file, finding list of relevant files and then changing all the dozen of project files. Which is hell of the work.
Even if it was implemented, how would this implementation decide which project file is a main one? (to get filelist from) And if your projects all have different lists of files? This would need merging etc. etc.
For now cocos new only copies files from one place to another. Frankly, Cocos2d-x have much more serious problems to solve, so I don't think this feature will ever be implemented.
However, you could customize project files/folders to your needs to make process of adding files more or less convenient and sometimes even automatic. Here are some clues:
It is easy to change Android and Linux Makefiles so they will pick up all the source files from a specified folder. Just use wildcards. Resource files are picked up by default.
For Visual Studio solutions (Win32, Win8, WinPhone) you could enable "Show all files" to see all files that either added or not. Unfortunately you can only see files down in the folder tree relative to project folder. To see Classes folder, make a symlink (link /j) of a Classes folder near .vcxproj file. You'd better don't copy/move project folder after it. And don't forget to add this linked folder to ignore list of your versioning system, or you will end up with duplicated files. Resource files are picked up by default for Windows projects.
I am not aware of any solution for XCode project, so you basically stuck with manual source file addition. Which is most annoying among all platforms. However it can pick up resource folders with all files.
Anyway, I would be glad if someone would proof me wrong and would write some kind of script to solve this problem (and also to change project name, company id, automatic versioning, etc.). I believe it could be done more or less reliably with, let's say Python and some regex magic. At least until project file formats will change.
I have spent considerable time to tune up Sublime Text 2 configs. Now I would want to share my configurations (one file) and installed packages list with my friend. What would be the best way to do this? Manually pick related folders in Packages and zip them or something else? Preferably I'd like to create an automated script which could be copy-pasted to my friend and others.
My friend is using Linux. I am using OSX, which may cause some extra problems.
If you are using Package Control, send your friend Package Control.sublime-settings. In fact, the easiest way to transfer settings between machines is to move the User package between machines. This, in combination with package control, makes setting up on a new machine relatively quick and painless. The only issues your friend might have is mismatched key binding or other system specific settings. These files have the notation some_name (platform).extension. Other than that, I wouldn't forsee any issues (unless you are using OS X specific plugins).
Package control has a good doc talking about how to sync Sublime settings and install packages: https://packagecontrol.io/docs/syncing
Using Windows
Open a new Command Prompt and type the following at the command line:
cd "C:\Users\<name>\AppData\Roaming\Sublime Text 3\Installed Packages" dir
Then copy and paste to an editor (e.g. Sublime Text) and filter out the relevant parts. Not great but working.
I often find the quote "InstallUtil.exe" is an ugly pattern or "Don't use InstallUtil.exe" and that I should use native WIX or Installation package patterns and I still don't understood why.
I stepped away from using InstallUtil to install a .NET service as I finally learnt that writing registry keys for such an action should be an un-install-able action - and I've come to terms with this as correct.
As I've been working through my WIX installer for a relatively complex product, I have found myself in need of creating or updating SQL Server databases, creating or updating IIS Applications and finally updating or creating configuration files.
Each of my components (features) are optional, but they all share the same configuration file. As my product uses unity, its important to note that this library contains strong support for reading/updating/removing components from the Unity Configuration block, therefore it seems fairly smart to me that I should take advantages of these blocks via Installation Components (i.e. InstallUtil) to create or update my configuration file at installation time.
Just to be clear here, my installer does not natively contain a configuration file for my application: at installation time, the installer has no idea as to the shape of it as its based on the features selected. Surely I should be embedding this knowledge into each of the modules that are to be deployed and not in the remit of the installer which is now a completely independent project? Wouldn't this break O-O principals even if we are talking about installation?
I'd really appreciate some guidance as to whether this is good practise or not? Am I reading 'InstallUtil' is bad for installing services, or is it that using 'InstallUtil' is bad full-stop? If so, what are my options for smart updating of configuration files?
The main reason for avoiding InstallUtil is that it runs outside of the installation transaction, so Windows Installer cannot keep track of what it's done.
I have used InstallUtil on a few occasions, when I just couldn't get Wix to do what I needed and didn't have time to write a custom action. In this case I called the InstallUtilLib version as I feel this is a cleaner approach.
I used the this blog as a guide as to how to achieve this.