I've got one KVM and some LUNs (Compellent SAN) in a multipath storage pool.
All filesystems are xfs.
> # virsh vol-list --pool multipath
dm-3 /dev/mapper/maildata-store-2-repl
dm-4 /dev/mapper/maildata-store-1-back
dm-5 /dev/mapper/metadata-store-2-repl
dm-6 /dev/mapper/metadata-store-1-back
dm-7 /dev/mapper/images
One LUN is dedicated for the storage of VMs (/var/lib/libvirt/images) and others will mounted directly in VMs for future storage of mail and relative metadata.
# df /dev/mapper/images1
Sys. de fichiers blocs de 1K Utilisé Disponible Uti% Monté sur
/dev/mapper/images1 209611780 18752452 190859328 9% /var/lib/libvirt/images
fio is used to compare IOPs on random write:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/tmp/10g.file --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
So I've got this result for the fio test while writing in tmp folder of a VM (/dev/mapper/images), quite good!
write: IOPS=66.1k, BW=258MiB/s
Now, I attach a LUN to this VM with this xml file:
<disk type='block' device='lun'>
<driver name='qemu' type='raw'/>
<source dev='/dev/mapper/maildata-store-1-back'/>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
And this command:
virsh attach-device VM_TEST --file lun.xml --persistent
Then, on VM_TEST:
#fdisk /dev/sda
#mkfs.xfs /dev/sda1
#mount /dev/sda1 /test
And rerun the fio test on the newly created partition:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=/test/10g.file --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
The results are quite worse:
write: IOPS=17.6k, BW=68.7MiB/s
I've played with different options in the xml file, like cache=none, bus=virtio, .., but I didn't manage to really increase the measures.
For now I'm stucked.
I don't really where to look for.
Thank you.
So, I manage to get same IOPs on hosts and guests with this tuning:
<driver name='qemu' type='raw' cache='directsync' io='native'/>
I also try to mount the block device as a disk or as a lun:
<disk type='block' device='lun'>
<target dev='sda' bus='scsi'/>
And
<disk type='block' device='disk'>
<target dev='sda' bus='virtio'/>
With quite the same results.
Related
I'm trying to import a huge dataset to ArangoDB via arangoimp. It's about 55 million edges. I alreasy increased the size of the wal.logfiles ( --wal.logfile-size to 1024k from 32k). This solved the last error. But now i get the following error:
WARNING {collector} got unexpected error in MMFilesCollectorThread::collect: no journal
ERROR cannot create datafile '/usr/local/var/lib/arangodb3/databases/database-1/collection-2088918365-385765492/temp-2153337069.db': Too many open files
Importstatement was: arangoimp --file links_de.csv --type csv --collection links
Is there a way to let arangoimp work more iterativ like doing chunks or something? I would be quite complicated to split the CSV in some parts because of its size...
thanks a lot!
i finally solved it:
Too many open files was the hint. You need to raise the limit for open file handles. So before starting arango on unix / mac os run ulimit -n 16384 or higher to raise the limit for this shell session. Afterwards the import still takes several hours but worked for me. You can also increase the journal size of the collection in arango to reduce the number of needed file handles.
Also see there: ArangoDB Too many open files
I have 70+ raster images in TIFF format that I am trying to merge.
Originals can be found here:
http://www.faa.gov/air_traffic/flight_info/aeronav/digital_products/vfr/
After pre-processing (pct2rgb, gdalwarp individual charts, gdal_translate to cut the collars) I try to run them through gdalwarp to mosaic them using a command like this:
gdalwarp --config GDAL_CACHEMAX 3000 -overwrite -wm 3000 -r bilinear -srcnodata 0 -dstnodata 0 -wo "NUM_THREADS=3" /data/aeronav/sec/c/Albuquerque_c.tif .....70 other file names ...master.tif
After 12 hours of processing:
Creating output file that is 321521P x 125647L.
Processing input file /data/aeronav/sec/c/Albuquerque_c.tif.
0...10...20...30...40...
This means gdalwarp is never going to finish.
In contrast. A gdal_merge command like this:
gdal_merge.py -n 0 -a_nodata 0 -o /data/aeronav/sec/master.tif /data/aeronav/sec/c/Albuquerque_c.tif ......70 plus files.....
Finishes in couple of hours.
Problem with gdal_merge is inferior quality output because of "average" sampling. I would like to use "bilinear" at the minimum - and "cubic" sampling if possible and for that gdalwarp is required.
Why is there such a big difference in performance of the two ? Why doesn't gdalwarp want to finish ? Is there any other command line option to speed things up in gadalwarp or is there a way to add sampling option to gdal_merge ?
It seems gdalwarp is not the ideal command to merge these GeoTiffs (since I am not interested in warping again). Instead I used
gdalbuildvrt /data/aeronav/sec/master.virt .... 70+ files in order
to build a virtual mosaic. And then I used gdal_translate to convert the virt file into a GeoTiff:
gdal_translate -of GTiff /data/aeronav/sec/master.virt /data/aeronav/sec/master.tif
That's it—this took less than an hour (even faster than gdal_merge and preserves quality of original files).
I am trying to remove identical lines in a file having 1.8 million records and create a new file. Using the following command:
sort tmp1.csv | uniq -c | sort -nr > tmp2.csv
Running the script creates a new file sort.exe.stackdump with the following information:
"Exception: STATUS_ACCESS_VIOLATION at rip=00180144805
..
..
program=C:\cygwin64\bin\sort.exe, pid 6136, thread main
cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B"
The script works for a small file with 10 lines. Seems like sort.exe cannot handle so many records. How do I work with such a large file with more than 1.8 million records? We do not have any database other than ACCESS and I was trying to do this manually in ACCESS.
It sounds like your sort command is broken. Since the path says cygwin, i'm assuming this is GNU sort, which generally should have no problem with this task, given sufficient memory and disk space. Try playing with flags to adjust where and how much it uses the disk: http://www.gnu.org/software/coreutils/manual/html_node/sort-invocation.html
The following awk command seemed to be a much faster way to get rid of the uniqe values:
awk '!v[$0]++' $FILE2 > tmp.csv
where $FILE2 is the file name with duplicate values.
I am playing with hortonworks sandbox to learn hadoop etc.
I am trying to load a file on a single machine "cluster":
A = LOAD 'googlebooks-eng-all-3gram-20090715-0.csv' using PigStorage('\t')
AS (ngram:chararray, year:int, count1:int, count2:int, count3:int);
B = LIMIT A 10;
Dump B;
Unfortunately the file is slightly too big for the ram that I have on my VM..
I am wondering if it's possible to LOAD a subset of the .csv file?
Is something like this possible:
LOAD 'googlebooks-eng-all-3gram-20090715-0.csv' using PigStorage('\t') LOAD ONLY FIRST 100MB?
Why exactly do you need to load the entire file into RAM? You should be able to run the whole file regardless of how much memory you need. Try adding this to the top of your script:
--avoid java.lang.OutOfMemoryError: Java heap space (execmode: -x local)
set io.sort.mb 10;
Your pig script will now read as:
--avoid java.lang.OutOfMemoryError: Java heap space (execmode: -x local)
set io.sort.mb 10;
A = LOAD 'googlebooks-eng-all-3gram-20090715-0.csv' using PigStorage('\t')
AS (ngram:chararray, year:int, count1:int, count2:int, count3:int);
B = LIMIT A 10;
Dump B;
Assuming you're just getting an OutOfMemoryError when you are running your script, this should solve your problem.
The way you define you solutions is not possible while in Hadoop however if you can achieve your objective when you are in OS Shell, rather than Hadoop shell. In Linux shell you can write a script to read first 100MB from source file, save it to local file system and then use as Pig source.
#Script .sh
# Read file and save 100 MB content in file system
# Create N files of 100MB each
# write a pig_script to process your data as shown below
# Launch Pig script and pass the N files as parameter as below:
pig -f pigscript.pig -param inputparm=/user/currentuser/File1.File2,..,FileN
#pigscript.pig
A = LOAD '$inputparm' using PigStorage('\t') AS (ngram:chararray, year:int, count1:int, count2:int, count3:int);
B = LIMIT A 10;
Dump B;
In general case, multiple files can be passed in Hadoop shell by their name, so you call out file names from Hadoop shell as well.
The key here is that in Pig there is no default way to read x from a file and process, it is all or nothing so you may need to find ways to solve achieve your objective.
Can anyone see anything wrong with the following set of commands? Every time I run these image.png is a image of the UK and not the JOSM map I exported. I'm guessing there's something awry with the db import however the output mentions that it's processing my coords and data.
Steps:
1 - Exported a .osm file from JOSM or Merkaator.
2 - Imported into psql using the following command:
osm2pgsql -m -d gis -S ~/mapnik/default.style -b 103,1.3,104,1.4 ion.osm -v -c
The output for this looks like:
marshall#ubuntu:~/mapnik$ osm2pgsql -m -d gis -S ~/mapnik/default.style -b 103,1.3,104,1.4 ion.osm -v -c
osm2pgsql SVN version 0.66-
Using projection SRS 900913 (Spherical Mercator)
Applying Bounding box: 103.000000,1.300000 to 104.000000,1.400000
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Mid: Ram, scale=100
Reading in file: ion.osm
Processing: Node(25k) Way(3k) Relation(0k)
Node stats: total(25760), max(844548651)
Way stats: total(3783), max(69993379)
Relation stats: total(27), max(536780)
Writing way(3k)
Writing rel(0k)
Committing transaction for planet_osm_point
Sorting data and creating indexes for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_roads
Sorting data and creating indexes for planet_osm_line
Committing transaction for planet_osm_polygon
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Completed planet_osm_polygon
Completed planet_osm_roads
Completed planet_osm_point
Completed planet_osm_line
I can see the correct lat/lon coords being passed in, I'm not sure how to verify this within the database
3 - ./generate_xml.py --accept-none --dbname gis --symbols ./symbols/ --world_boundaries ../world_boundaries/
4 - ./generate_image.py
At this point image.png is a map of the UK, not Singapore which I have specified.
Can anyone see anything wrong with this? This is with mapnik 0.71 on ubuntu
Found the solution.
The issue is that the generate_image.py script does not read the data from the database but rather has it hardcoded inside. I'm not sure the reasoning behind this.
The solution is to edit generate_image.py manually and change the relevant line:
ll = (103,1.3,104,1.4)