I have a bunch of point shapefiles, each ~2-3GB, which I upload into GRASS GIS, to find the diversity of points in each cell of a gridded shapefile using 'v.vect.stats'.
This operation worked fine for smaller datasets(<2 GB). I'm not sure if it's because of the size, but I now get this message in the console:
v.vect.stats: spindex_rw.c:1362: rtree_get_node: Assertion `t->nb[level][which].n.level == level' failed.
Followed by:
Aborted (core dumped)
I'd be grateful for any ideas on what could be causing this and how I can get around it.
EDIT: The above works on the same shapefile when I reduce the number of points in it. So I'm guessing this is an issue with the size of the file. Unfortunately, I cannot use a shapefile with reduced points because each point in a shapefile has a unique ID, so calculating the diversity of points needs to happen with all the points.
I tried setting GRASS_VECTOR_LOWMEM=1, so that GRASS uses less memory for the v.vect.stats operation, but this does not help. I get the same 'Aborted (core dumped)' message.
Related
I am working on reproducing the results reported in this paper. A UNET based network is used for estimating sound speed map from raw Ultrasound channel data. I have been stuck in further reducing the train/val loss for a long time.
Basically, I followed their methods of data simulation, preprocessing and used the same network architecture, hyperparameters (include kernel initializer, batch size, decay rate, etc.). The input size is 128*1024 rather than 192*2048 according to my ultrasound probe (based on their recent paper, the input size won't affect the performance).
So my question is do you have any suggestion to further investigate this problem based on your experience?
I attached my results (loss curves and images).
RMSE loss
estimated sound speed
And their results
RMSE loss
estimated sound speed
It seems my network failed to have a comparative convergence at the background, that could explain that I got a larger initial loss.
PS: Unfortunately, the paper didn't provide codes, so I have no clue to some details in terms of data simulation and training. I have contacted the author but hasn't got response yet.
The author mentioned somewhere instead of using a pixel-wise MSE, try a larger window size 3*3 or 5*5, I am not clear whether it is used for training or metric evaluation, any reference for the former?
enter image description hereI encountered one perplexing situation: I trained a Transformer model for conditional music generation, finding the training loss decrease while the validation loss keeps increasing from the beginning, which is quite strange.
I attached a tensorboard screenshot for your reference. Here is the data allocation of about 2.7k songs: train:valid:test = 7:2:1.
I even try changing the learning rate from 0.01 to 0.00001, resulting nonetheless the same situation. On this above, shall I use more data for training or are there any other possible solutions regarding this?
I was trying to run fcn on my data in caffe. I was able to convert my image sets into lmdb by convert_imageset builtin function caffe. However, once I wanted to train the net, it gave me the following error:
Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
.....
Aborted (core dumped)
I went through many online resources to solve the memory failure, but most of them suggesting reducing batch size. Even, I reduced the size of images to 256x256. I could not tackle this issue yet.
I checked the memory of GPU by this command nvidia-smi, and the model is Nvidia GT 730 and the memory is 1998 MiB. Since the batch size in train_val.prototxt is 1, I can not do anythin in train_val.prototxt. So my questions are:
By looking at log file in Terminal,I realized that whenever convert_imageset converting the data into LMDB, it is taking 1000 image in a group. Is it possible I change this number in line 143 and 151 of convert_imageset.cpp to a smaller (for example 2; to take two image at a time), recompile caffe, and then convert images to lmdb by using convert_imageset? Does it make sense?
If the answer to question 1 is yes, how can I compile caffe again,
should I remove build folder and again do caffe installation from
scratch?
How caffe process the LMDB data? Is it like taking a batch of those 1000 images showing while running convert_imagenet?
Your help is really appreciated.
Thanks...
AFAIK, there is no effect for the number of entries committed to lmdb at each transaction (txn->Commit();) on the cuda out of memory.
If you do want to re-compile caffe for whatever reason, simply run make clean. This will clear everything and let you re-compile from scratch.
Again, AFAIK, caffe access lmdb batch_size images at a time regardless of the size of the transaction size used when writing the dataset.
Are you sure batch_size is set to 1 for both TRAIN and TEST phases?
Seems this related PR is dead now, is there any workaround to use early stopping in Caffe? maybe using python on top of Caffe?
A first part is easy to do manually: let monitor your validation error then stop when this one do not change a lot (below a threshold). Then let consider the state with the lowest validation error as the "optimal" network.
The real problem is then to benefit from the full train+val dataset from there. There are two basic strategies:
Retrain your network with train+val for the same number of epoch OR for the same number of data (i.e let compute the number of minibatches that were used to reach the "optimal" state and set the number of passes such that the same number of minibatches (with same size...) go through the network
Let keep the "optimal" network then add the validation data and continue the training. If you reach the same error rate as before, let stop. Else, let just fix an a priori number of epochs.
You could apply this patch for early stopping to standard Caffe RC 1.0.0. It adds an optional early_stop_param into the solver. You can specify the test network ID, the length of tries to check for improvement in the test loss and a "skip" so that not every iteration is tried. Disclosure: I am one of the developers.
So I have been making a simple HTML5 tuner using the Web Audio API. I have it all set up to respond to the correct frequencies, the problem seems to be with getting the actual frequencies. Using the input, I create an array of the spectrum where I look for the highest value and use that frequency as the one to feed into the tuner. The problem is that when creating an analyser in Web Audio it can not become more specific than an FFT value of 2048. When using this if i play a 440hz note, the closest note in the array is something like 430hz and the next value seems to be higher than 440. Therefor the tuner will think I am playing these notes when infact the loudest frequency should be 440hz and not 430hz. Since this frequency does not exist in the analyser array I am trying to figure out a way around this or if I am missing something very obvious.
I am very new at this so any help would be very appreciated.
Thanks
There are a number of approaches to implementing pitch detection. This paper provides a review of them. Their conclusion is that using FFTs may not be the best way to go - however, it's unclear quite what their FFT-based algorithm actually did.
If you're simply tuning guitar strings to fixed frequencies, much simpler approaches exist. Building a fully chromatic tuner that does not know a-priori the frequency to expect is hard.
The FFT approach you're using is entirely possible (I've built a robust musical instrument tuner using this approach that is being used white-label by a number of 3rd parties). However you need a significant amount of post-processing of the FFT data.
To start, you solve the resolution problem using the Short Timer FFT (STFT) - or more precisely - a succession of them. The process is described nicely in this article.
If you intend building a tuner for guitar and bass guitar (and let's face it, everyone who asks the question here is), you'll need t least a 4092-point DFT with overlapping windows in order not to violate the nyquist rate on the bottom E1 string at ~41Hz.
You have a bunch of other algorithmic and usability hurdles to overcome. Not least, perceived pitch and the spectral peak aren't always the same. Taking the spectral peak from the STFT doesn't work reliably (this is also why the basic auto-correlation approach is also broken).