I`m trying to create map which has several layers. One of them consist of country borders (admin_level = 2,3), country name and region names. I use data of coordinates of Armenia and China from https://download.geofabrik.de/asia.html. I succesfully consolidated these countries with Osmosis 0.40.1.:
osmosis --read-pbf armenia-latest.osm.pbf --read-pbf china-latest.osm.pbf --merge --write-pbf ArChina.osm.pbf
And extracted border with admin_level = 2,3:
osmosis --read-pbf ArChina.osm.pbf --tf accept-relation boundary=administrative admin_level=2,3 --tf reject-ways admin_level=5,6,7,8,9 --used-way idTrackerType=Dynamic --tf reject-nodes admin_level=4,5,6,7,8,9 --tf reject-nodes place=village,town,suburb,city --used-node idTrackerType=Dynamic --write-xml ArChinaBord1.osm
Then I converted it to .osm.pbf with Osmosis.
I understand that I didn't remove all the nodes such as bus stops, natural objects, etc. That`s not the problem.
The problem is - when I`ve launched this map on Ubuntu server - I saw the borders of 2 countries, region names of both of them and only 1 country name (Armenia). I was trying to zoom in and zoom out, but it didn't help. Сhina name is not showing at all.
I checked ArChinaBord1.osm (consolidated file with Armenia and China borders, names). The xml structure of Armenia and China names (node with place=country) is the same, but only one name is shown.
Could you give me some advice, how it could be fixed?
I have Snakefile as following:
SAMPLES, = glob_wildcards("data/{sample}_R1.fq.gz")
rule all:
input:
expand("samtools_sorted_out/{sample}.raw.snps.indels.g.vcf", sample=SAMPLES),
expand("samtools_sorted_out/combined_gvcf")
rule combine_gvcf:
input: "samtools_sorted_out/{sample}.raw.snps.indels.g.vcf"
output:directory("samtools_sorted_out/combined_gvcf")
params: gvcf_file_list="gvcf_files.list",
gatk4="/storage/anaconda3/envs/exome/share/gatk4-4.1.0.0-0/gatk-package-4.1.0.0-local.jar"
shell:"""
java -DGATK_STACKTRACE_ON_USER_EXCEPTION=true \
-jar {params.gatk4} GenomicsDBImport \
-V {params.gvcf_file_list} \
--genomicsdb-workspace-path {output}
"""
When I test it with dry run, I got error:
RuleException in line 335 of /data/yifangt/exomecapture/Snakefile:
Wildcards in input, params, log or benchmark file of rule combine_gvcf cannot be determined from output files:
'sample'
There are two places that I need some help:
The {output} is a folder that will be created by the shell part;
The {output} folder was hard-coded manually required by the command line (and the contents are unknown ahead of time).
The problem seems to be with the {output} without expansion as compared with the {input} which does.
How should I handle with this situation? Thanks a lot!
I've gone through Mike Bostock's excellent tutorials on Command-Line Cartography and I'm confused by his use of his ndjson-split utility. That program is used to split up an array of objects in a json file, putting each object in the array on a single line. (Reference: https://github.com/mbostock/ndjson-cli)
In Part Two of the tutorial (https://medium.com/#mbostock/command-line-cartography-part-2-c3a82c5c0f3#.624i8b4iy) Mike uses ndjson-split on a geojson file:
ndjson-split 'd.features' \
< ca-albers.json \
> ca-albers.ndjson
He explains:
The output here looks underwhelmingly similar to the ca-albers.json we
saw previously; the only difference is that there is one feature (one
census tract) per line.
However, it seems there is another big difference. The new file does not contain all of the data that was in the original file. Specifically, the start of the original JSON object, {"type":"FeatureCollection" ... is gone.
Mike doesn't explain why this additional key is not needed in the geojson file (the resulting files work perfectly).
Anyone know why? Is this key not needed for valid geoJSON?
What I am starting with, is the postcode table from the netherlands. I split it up into a couple of csv files, containing for instance the city as subject, PartOf as predicate and municipality as object. This gives you this in a file:
city,PartOf,municipality
Meppel,PartOf,Meppel
Nijeveen,PartOf,Meppel
Rogat,PartOf,Meppel
Now I would like to get this data into MarkLogic. And I can import csv-files, I can import triples, but I can't figure out the combination.
I would suggest rewriting it slightly so it conforms to the N-Triples format, giving it the .nt extension, and then using MLCP to load it as input_type rdf.
HTH!
You can use Google Refine to convert CSV data to RDF. After that, MLCP can be used to push that data. You can do something like this -
$ mlcp.sh import -username user -password password -host localhost \
-port 8000 -input_file_path /my/data -mode local \
-input_file_type rdf
For more on loading triples using MLCP you can refer this MarkLogic Community Page
So I have a json file that has multiple records, looking like this...
{"2597401":[{
"jobID":"2597401",
"account":"TG-CCR120014",
"user":"c",
"pkgT":{"pgi/7.2-5":{"libA":["libpgc.so"],"flavor":["default"]}},
"startEpoch":"1338497979",
"runTime":"1022",
"execType":"user:binary",
"exec":"/share/home/01482/c/appker/ranger/NPB3.3.1/NPB3.3-MPI/bin/ft.D.64",
"numNodes":"4",
"sha1":"5a79879235aa31b6a46e73b43879428e2a175db5",
"execEpoch":1336766742,
"execModify":"Fri May 11 15:05:42 2012",
"startTime":"Thu May 31 15:59:39 2012",
"numCores":"64",
"sizeT":{"bss":"1881400168","text":"239574","data":"22504"}},
{"jobID":"2597401",
"account":"TG-CCR120014",
"user":"c",
"pkgT":{"pgi/7.2-5":{"libA":["libpgc.so"],"flavor":["default"]}},
"startEpoch":"1338497946",
"runTime":"33",
"execType":"user:binary",
"exec":"/share/home/01482/c/appker/ranger/NPB3.3.1/NPB3.3-MPI/bin/cg.C.64",
"numNodes":"4",
"sha1":"caf415e011e28b7e4e5b050fb61cbf71a62a9789",
"execEpoch":1336766735,
"execModify":"Fri May 11 15:05:35 2012",
"startTime":"Thu May 31 15:59:06 2012",
"numCores":"64",
"sizeT":{"bss":"29630984","text":"225749","data":"20360"}},
{"jobID":"2597401",
"account":"TG-CCR120014",
"user":"c",
"pkgT":{"pgi/7.2-5":{"libA":["libpgc.so"],"flavor":["default"]}},
"startEpoch":"1338500447",
"runTime":"145",
"execType":"user:binary",
"exec":"/share/home/01482/appker/ranger/NPB3.3.1/NPB3.3-MPI/bin/mg.D.64",
"numNodes":"4",
"sha1":"173de32e1514ad097b1c051ec49c4eb240f2001f",
"execEpoch":1336766756,
"execModify":"Fri May 11 15:05:56 2012",
"startTime":"Thu May 31 16:40:47 2012",
"numCores":"64",
"sizeT":{"bss":"456954120","text":"426186","data":"22184"}},
{"2597401":[{....
Each record is on a single line. So the general format of the json file is as follows:
{"1111111: [{"jobID":value,"account":value,"user":value,"pkgT":{value:{"libA":[VALUES],"flavor":[value]}},"startEpoch":value,"runTime":value,"execType":value,"exec":value,"numNodes":value,"sha1":value,"execEpoch":value,"execModify":value,"startTime":value,"numCores":value,"sizeT":{"bss":value,"text":value,"data":value}}
I am trying to use mongoimport to import these into a database using the following command:
mongoimport --db dbName --collection collectionName fileName --jsonArray
I read that each record should be on it's own line, which it is. However, when I import the file it imports it as a single document (aka record). How can I get mongo to interpret this file as containing multiple records instead of a single huge document? Thanks for your help!!!
I was able to fix it. I had curly brackets surrounding all the text in the file. When I removed them, it successfully imported the records individually.