I am trying to run a sql query with a 50 GB CSV file but my GPU Memory is of only 40GB. How can I do the processing?
Also I am only able to run blazingsql with the jupyter notebook available with their docker image, can anyone please help me how to install it locally?
As it is not being possible with the conda command available on their github.
One way to do this today is to use Dask-SQL. Because it's built on Dask, Dask-SQL inherits Dask's ability to handle larger-than-memory workloads.
The easiest way to install Dask-SQL and use GPUs is to create a conda environment or pull a Docker container using the RAPIDS release selector.
Related
I am doing some load tests and according to my observations seems the cos-stable machine underperforms a normal linux machine.
I have started up the cos-stable machine with a container running node-alpine linux with a js application and execute a load test, then I did the same using an ubuntu machine, installing the node on it to run the same js application.
Both OS' were using the same resources, n1 machines with 2vcpu, 8Gram and 10G SSD disk.
Does anyone have information about how to tune the cos-stable container to have the same performance as I had in the ubuntu machine?
Thanks.
well according to the comment of this user in the docker forum
I have to agree with the others, I always use official images if I can and I always select the Alpine version if available. Why use Alpine? Two reasons:
Much small images. Ubuntu is 188MB alone. Then you add your app on top
of that probably exceeding 200MB. Alpine Linux is only 4MB! After
adding my Python runtime and code most of my images are only 52MB.
Compare that will almost 200MB of Ubuntu. Smaller images are smaller
upload/download and take up less disk space.
In adding to that, if you are running the same application in both machines with different OS you have to consider pros and cons of using that OS
if you want to customize your Container Optimized OS VM creation you can follow the instructions in this link
I hope this information can be useful.
I want to use caffe in Google colab for experiment purpose, and my understanding is that i need to setup and install caffe every time i open python notebook in every runtime. This is very hard and time consuming because caffe takes lot of time to install and sometimes fails to install.
So i wanted to know if i am doing something wrong or is that the way how colab runtime works?
I would like to know of any other alternative platform for free which i can use if i need to setup caffe every time i use colab. I want to use Alex net pre tained caffe model
And moreover i wasted last 2 complete days setting up caffe locally on windows 10 machine, which by the way still not working, i am really getting sick of this caffe.
On Colab you can install the pre-compiled Caffe using
!apt install caffe-cpu
For GPU runtime, you can use caffe-cuda as well.
See instruction here.
is that the way how colab runtime works?
Yes it is!
To have persistent data you have to save your notebook to your drive:
File/Save a copy in Drive
I am beginner. I want install freebsd on VM and test open source world! I want to write a small function and to put it into kernel of OS, and then I want to use it in another program as a system function.
I just installed freebsd11 on VM. I know a command line environment! I should use a GUI? Where is kernel?
FreeBSD is a wonderful beast once you know all its capabilities, if you want to play with jails, ZFS and build your own kernels, probably this already build image can be a good starting point:
https://fabrik.red/post/test/ more info can be found here including scripts about how to create your own images/jails (custom kernel): https://github.com/fabrik-red/images
There is no GUI on the images, and maybe you will never need one, unless you want to use FreeBSD as a desktop, but FreeBSD shines more on headless systems (no GUI).
Update:
For GUI probably you can give a try to trueos.org or you can install your own Xorg, desk environments
Sounds like you should learn a bit more about what the operating system does before trying to modify the kernel. There are lots of resources about the unix kernels. See the developers handbook, https://www.freebsd.org/doc/en/books/developers-handbook/ or just google and you will find many resources.
I have a desktop without a GPU, on which I would like to develop code; and a machine on some cluster which has a GPU, and CUDA installed, but where I really can't "touch" anything and on which I won't run an IDE etc. I don't have root on any of the machines, woe is me.
So, essentially, I want to be able to compile and build my CUDA code on my own GPU-less desktop machine, then just copy it and test it on the other machine.
Can this be done despite the two hindering factors: I seem to recall the CUDA installer requiring the presence of a GPU; playing with the kernel; and doing other root-y stuff.
Notes:
I'll be using the standalone installer, not a package.
I'm on Fedora 22 with an x86_64 CPU.
Assuming you want to develop codes that use the CUDA runtime API, you can install the cuda toolkit on a system that does not have a GPU. Using the runfile installer method, simply answer no when prompted to install the driver.
If you want to compile codes (successfully) that use the CUDA driver API, that process will require a libcuda.so on your machine. This file is installed by the driver installer. There are various methods to "force" the driver installer to run on a machine without a GPU. You can get started by extracting the driver runfile installer (or downloading it separately) and passing the --help command line switch to the installer to learn about some of the options.
These methods will not allow you to run those codes on a machine with no GPU of course. Furthermore, the process of moving a compiled binary from one machine to another, and expecting it to run correctly, is troublesome in my opinion. Therefore my suggestion would be to re-compile the code on a target machine. Otherwise getting a compiled binary to run from one machine to the next is a question that is not unique to CUDA, and is outside the scope of my answer.
If you have no intention of running the codes on the non-GPU machine, and are willing to recompile on the target machine, then you can probably develop driver API codes even without libcuda.so (or there is a libcuda.so stub that you could try linking against just for compilation-test purposes, which is installed by the CUDA installer, if you search for it: /usr/local/cuda/lib64/stubs). If you don't link your driver API code against -lcuda, then you'll get a link error of course, but it should not matter, given the previously stated caveats.
Fedora 22 is not officially supported by CUDA 7.5 or prior. YMMV.
If you don't run the driver installer, you don't need to be a root user for any of this. Of course the install locations you pass to the installer must be those that your user privilege allows access to.
We have a Windows 8 Modern / Metro app that we wish to deploy outside the store. I have read a bunch of materials on getting a side loading license to allow this. However, we need to go beyond just copying the app over and running some PowerShell commands, we also need to install a particular hardware driver and make some registry keys.
Can anyone recommend an installer creator that can perform all of these tasks?
Out of curiosity, is it possible to distribute through the Windows Store and have it do more than just copy the app down, i.e. execute an installations script?
By design, the installation process for a Store app, whether downloaded from the Store or side-loaded, is not customizable. For Store downloads, this is done to guarantee that the app install won't create anything that cannot be reversed during an uninstall, e.g. create registry entries, copy files, and otherwise do things that can permanently alter the system.
For side-loading, although you cannot alter the specific package deployment process itself, you can use the Powershell deployment as just one step in an overall install process. You should be able to build such an installer with any number of existing Desktop application installer tools, which give you control over all steps in your install process including registry, drivers, running scripts, etc. So you'd just have the app package and PS scripts as part of the installer .msi or .exe, and executing the PS script is done from the installer.
(By the way, the Store/PS app install processes use the same API in Windows.Management.Deployment.PackageManager to do the work. This API can be used only from a full-trust process like the Store (a piece of system code) or Powershell running in an admin console, but is there so desktop apps can implement alternate install procedures directly.)