How can I convert a json in geojson format? - json

I have seen other similar questions and with good answers, only that the interntarlo in my program tells me error, I may be wrong, the process is as follows. And I think that is correct.
from sys import argv
from os.path import exists
import json
jdata({"metadata":{"resultset":
{"offset":1,"count":125210,"limit":25}},"results":
[{"elevation":139,"mindate":"1948-01-01","maxdate":"2014-01-
01","latitude":31.5702,"name":"ABBEVILLE, AL US","datacoverage":0.8813,"id":"COOP:010008","elevationUnit":"METERS","longitude":-85.2482},{"elevation":249.3,"mindate":"1938-01-01","maxdate":"2015-11-01","latitude":34.2553,"name":"ADDISON, AL US","datacoverage":0.5059,"id":"COOP:010063","elevationUnit":"METERS","longitude":-87.1814},{"elevation":302.1,"mindate":"1940-05-01","maxdate":"1962-03-01","latitude":34.41667,"name":"ADDISON CENTRAL TOWER, AL US","datacoverage":0.9658,"id":"COOP:010071","elevationUnit":"METERS","longitude":-87.31667},{"elevation":172.2,"mindate":"1995-04-01","maxdate":"2015-11-01","latitude":33.17833,"name":"ALABASTER SHELBY CO AIRPORT, AL US","datacoverage":0.8064,"id":"COOP:010116","elevationUnit":"METERS","longitude":-86.78167},{"elevation":183.8,"mindate":"1949-01-01","maxdate":"1949-12-01","latitude":34.6891,"name":"BELLE MINA, AL US","datacoverage":1,"id":"COOP:010117","elevationUnit":"METERS","longitude":-86.8819},{"elevation":34.1,"mindate":"1935-05-01","maxdate":"1936-11-01","latitude":31.13333,"name":"ALAGA, AL US","datacoverage":0.2624,"id":"COOP:010125","elevationUnit":"METERS","longitude":-85.06667},{"elevation":53.3,"mindate":"1940-11-01","maxdate":"2014-12-01","latitude":32.2322,"name":"ALBERTA, AL US","datacoverage":0.9888,"id":"COOP:010140","elevationUnit":"METERS","longitude":-87.4104},{"elevation":348.1,"mindate":"1931-01-01","maxdate":"1977-06-01","latitude":34.23333,"name":"ALBERTVILLE, AL US","datacoverage":0.9535,"id":"COOP:010148","elevationUnit":"METERS","longitude":-86.16667},{"elevation":195.1,"mindate":"1969-10-01","maxdate":"2015-11-01","latitude":32.9452,"name":"ALEXANDER CITY, AL US","datacoverage":0.9946,"id":"COOP:010160","elevationUnit":"METERS","longitude":-85.948},{"elevation":200.9,"mindate":"1942-11-01","maxdate":"1969-10-01","latitude":32.98333,"name":"ALEXANDER CITY 6 NE, AL US","datacoverage":0.9629,"id":"COOP:010163","elevationUnit":"METERS","longitude":-85.86667},{"elevation":59.4,"mindate":"1940-01-01","maxdate":"2015-11-01","latitude":33.1272,"name":"ALICEVILLE, AL US","datacoverage":0.9144,"id":"COOP:010178","elevationUnit":"METERS","longitude":-88.155},{"elevation":50.3,"mindate":"1980-05-01","maxdate":"2015-11-01","latitude":33.21,"name":"ALICEVILLE L AND D, AL US","datacoverage":0.9883,"id":"COOP:010184","elevationUnit":"METERS","longitude":-88.2878},{"elevation":76.2,"mindate":"1938-01-01","maxdate":"2015-11-01","latitude":31.3071,"name":"ANDALUSIA 3 W, AL US","datacoverage":0.9744,"id":"COOP:010252","elevationUnit":"METERS","longitude":-86.5226},{"elevation":208.8,"mindate":"2004-07-01","maxdate":"2015-11-01","latitude":34.9093,"name":"ANDERSON, AL US","datacoverage":1,"id":"COOP:010260","elevationUnit":"METERS","longitude":-87.2746},{"elevation":220.1,"mindate":"1948-01-01","maxdate":"1949-03-01","latitude":33.66667,"name":"ANNISTON, AL US","datacoverage":1,"id":"COOP:010267","elevationUnit":"METERS","longitude":-85.83333},{"elevation":181.1,"mindate":"1948-01-01","maxdate":"2015-11-01","latitude":33.5872,"name":"ANNISTON METROPOLITAN AIRPORT, AL US","datacoverage":0.9816,"id":"COOP:010272","elevationUnit":"METERS","longitude":-85.8556},{"elevation":227.1,"mindate":"1938-01-01","maxdate":"1983-10-01","latitude":34.06667,"name":"ARLEY 1 S, AL US","datacoverage":0.9691,"id":"COOP:010338","elevationUnit":"METERS","longitude":-87.23333},{"elevation":311.5,"mindate":"1948-01-01","maxdate":"2013-11-01","latitude":33.2941,"name":"ASHLAND, AL US","datacoverage":0.8872,"id":"COOP:010369","elevationUnit":"METERS","longitude":-85.7788},{"elevation":180.1,"mindate":"1941-06-01","maxdate":"1973-06-01","latitude":33.85,"name":"ASHVILLE, AL US","datacoverage":0.9429,"id":"COOP:010377","elevationUnit":"METERS","longitude":-86.33333},{"elevation":210,"mindate":"1948-01-01","maxdate":"2015-11-01","latitude":34.7752,"name":"ATHENS, AL US","datacoverage":0.5632,"id":"COOP:010390","elevationUnit":"METERS","longitude":-86.9508},{"elevation":219.5,"mindate":"1955-10-01","maxdate":"1991-07-01","latitude":34.8,"name":"ATHENS 2, AL US","datacoverage":0.9442,"id":"COOP:010395","elevationUnit":"METERS","longitude":-86.98333},{"elevation":220.1,"mindate":"1958-05-01","maxdate":"1958-06-01","latitude":34.8,"name":"ATHENS COURTHOUSE, AL US","datacoverage":0.998,"id":"COOP:010398","elevationUnit":"METERS","longitude":-86.96667},{"elevation":91.4,"mindate":"1940-01-01","maxdate":"2015-11-01","latitude":31.182,"name":"ATMORE, AL US","datacoverage":0.7103,"id":"COOP:010402","elevationUnit":"METERS","longitude":-87.439},{"elevation":67.1,"mindate":"1965-01-01","maxdate":"1982-10-01","latitude":31.01667,"name":"ATMORE, AL US","datacoverage":0.967,"id":"COOP:010407","elevationUnit":"METERS","longitude":-87.51667},{"elevation":223.1,"mindate":"1931-01-01","maxdate":"1970-12-01","latitude":32.6,"name":"AUBURN, AL US","datacoverage":0.9917,"id":"COOP:010422","elevationUnit":"METERS","longitude":-85.5}]}
script, in_file, out_file = argv
data = json.loads(jdata)
geo_map = {"type":"FeatureCollection"}
It is here that will not let me continue
geojson = {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry" : {
"type": "Point",
"coordinates": [point["lon"], point["lat"]],
},
"properties" : {
"mindate":[],
"maxdate":[],
"name":[],
"datacoverage":[],
"id":[],
"elevationUnit":[],
}
for Point in data]
output = open(out_file, 'w')
json.dump(geojson, output)
print geojson
My English is not very good, i hope you understand and can help me please.

Related

How to find json data?

chitown88 help me to find the json on this website : https://www.iwc.com/fr/fr/watch-collections/pilot-watches/iw329303-big-pilots-watch-43.html
It seems that you need to replace html by .productinfo.FR.json
Source : How to scrape specific information on a website
I would like to do the same output with this page : https://www.omegawatches.com/fr-fr/watch-omega-constellation-quartz-27-mm-12315276005001
But I cannot manage to scrape those informations because the page is dynamic and I cannot find the json data, I searched for hours.
Do you have any solutions in order to scrape the same output than the question source ?
There isn't anything special about this page. Just grab the right information with beautifulsoup:
import requests
from bs4 import BeautifulSoup
url = "https://www.omegawatches.com/fr-fr/watch-omega-constellation-quartz-27-mm-12315276005001"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
title = soup.title.text.split("|")[0].strip()
description = soup.select_one(".description-content").get_text(strip=True)
price = soup.select_one(".price").text
info = "\n".join(
tag.get_text(strip=True)
for tag in soup.select(
".product-info-details-right .li, .product-info-details-right li"
)
)
print(title)
print()
print(description)
print("-" * 80)
print(price)
print("-" * 80)
print(info)
Prints:
Constellation Quartz 27 mm - 123.15.27.60.05.001
L'esthétique particulièrement remarquable et intemporelle de la collection OMEGA Constellation se caractérise par l'originalité du cadran et la présence des fameuses griffes.Ce modèle brossé se distingue par un cadran en nacre blanche protégé par un verre saphir résistant aux rayures. La lunette sertie de diamants est montée sur un boîtier de 27 mm en acier inoxydable sur un bracelet également en acier inoxydable.Cette montre est animée par le calibre OMEGA 1376, un mouvement de précision à quartz.
--------------------------------------------------------------------------------
5 000,00 €
--------------------------------------------------------------------------------
Diamants
Entre‑corne :18 mm
Bracelet :acier
Boîtier :acier
Diamètre du boîtier :27 mm
Couleur du cadran :blanc
Verre :verre saphir bombé résistant aux rayures, traité antireflet à l’intérieur
Étanchéité :10 bars (100 mètres / 330 pieds)
Type de mouvement :quartz
Calibre :OMEGA 1376
Mouvement de précision à quartz, finition rhodiée.
Durée de vie de la pile :48 mois
Type :quartz
Swiss Made
GARANTIE 5 ANS
Paiement sécurisé
Livraison et retour offerts

Failure in Deserialization of File (NewtonSoft.JSON & VB.NET) [duplicate]

This question already has answers here:
How can I parse a JSON string that would cause illegal C# identifiers?
(3 answers)
Cannot deserialize the JSON array (e.g. [1,2,3]) into type ' ' because type requires JSON object (e.g. {"name":"value"}) to deserialize correctly
(6 answers)
Closed 1 year ago.
I have a small test file I'm trying to work with to deserialize the data into a model. I've already this using tokens in NewtonSoft.JSON but would rather do this properly with a model.
Example JSON
[
{
"faction": "Britain",
"id": "aa_barrage",
"image": "aa_barrage.png",
"import_id": "02",
"kredits": 1,
"rarity": "Standard",
"set": "Base",
"text": {
"de-DE": "Die gewählte Lufteinheit muss sich zurückziehen. Gibt deinem HQ +2 Verteidigung.",
"en-EN": "Target air unit must retreat. Give your HQ +2 defense.",
"es-ES": "La unidad aérea seleccionada debe retirarse. Otorga un +2 de defensa a tu cuartel general.",
"fr-FR": "L'unité aérienne visée doit battre en retraite. Confère +2 de défense à votre QG",
"it-IT": "L'unità aerea bersaglio deve Ritirarsi. Dai al tuo QG +2 difesa.",
"pl-PL": "Wybrana jednostka powietrzna musi się wycofać. Zapewnij swojemu sztabowi +2 do obrony.",
"pt-BR": "A unidade aérea na mira deve recuar. Dê +2 de defesa ao seu QG.",
"ru-RU": "Выбранный воздушный отряд должен отступить. Ваш штаб получает +2 к защите.",
"zh-Hans": "使一个空军撤退,你的总部获得 +2 防御力。",
"zh-Hant": "使一個空軍撤退,你的總部獲得 +2 防禦力。"
},
"title": {
"de-DE": "FLAK-SPERRFEUER",
"en-EN": "AA BARRAGE",
"es-ES": "BOMBARDEO AA",
"fr-FR": "BARRAGE AA",
"it-IT": "CONTRAEREA",
"pl-PL": "ZAPORA PRZECIWLOTNICZA",
"pt-BR": "BARRAGEM AA",
"ru-RU": "ЗЕНИТНЫЙ ОГОНЬ",
"zh-Hans": "防空弹幕",
"zh-Hant": "防空彈幕"
},
"type": "order"
},
{
"faction": "Britain",
"id": "active_sonar",
"image": "active_sonar.png",
"import_id": "fy",
"kredits": 0,
"rarity": "Elite",
"set": "OnlySpawnable",
"text": {
"de-DE": "Setzt alle feindlichen Einheiten fest. Karten in der gegnerischen Hand kosten 3 mehr.",
"en-EN": "Pin all enemy units. Increase cost of all cards in enemy hand by 3.",
"es-ES": "Inmoviliza todas las unidades enemigas. Las cartas en la mano del enemigo cuestan 3 más.",
"fr-FR": "Immobilise toutes les unités ennemies. Les cartes dans la main de l'ennemi coûtent 3 de plus.",
"it-IT": "Blocca tutte le unità nemiche. Aumenta di 3 il costo delle carte nella mano del nemico.",
"pl-PL": "Przyszpil wszystkie wrogie jednostki. Koszt kart w ręce przeciwnika wzrasta o 3.",
"pt-BR": "Imobilize todas as unidades inimigas. Todas as cartas na mão do inimigo custam mais 3.",
"ru-RU": "Блокирует все вражеские отряды. Карты в руке противника стоят на 3 больше.",
"zh-Hans": "压制敌方所有单位。对手所有手牌花费 +3。",
"zh-Hant": "壓制敵方所有單位。對手所有手牌花費 +3。"
},
"title": {
"de-DE": "AKTIVES SONAR",
"en-EN": "ACTIVE SONAR",
"es-ES": "SONAR ACTIVO",
"fr-FR": "SONAR ACTIF",
"it-IT": "SONAR ATTIVO",
"pl-PL": "AKTYWNY SONAR",
"pt-BR": "SONAR ATIVO",
"ru-RU": "АКТИВНЫЙ СОНАР",
"zh-Hans": "主动声呐",
"zh-Hant": "主動聲納"
},
"type": "order"
}
]
The current code I have for this is the following:
Module Program
Sub Main(args As String())
dim json As string = File.ReadAllText("C:\Temp\cards.json")
dim result As cards = JsonConvert.DeserializeObject(Of cards)(json)
End Sub
End Module
Public Class cards
public property cards as list(of card)
End Class
Public Class card
public property faction as string
public property id as string
public property image as string
public property import_id as string
public property kredits as integer
public property rarity as string
<JsonProperty("set")>
public property cardset as string
public property type as string
End Class
I can deserialize just one of these on its own, however when I try and deserialize with both it fails due to the following:
Newtonsoft.Json.JsonSerializationException: Cannot deserialize the current JSON array (e.g. [1,2,3]) into type 'json_deserialize_test.cards' because the type requires a JSON object (e.g. {"name":"value"}) to deserialize correctly.
To fix this error either change the JSON to a JSON object (e.g. {"name":"value"}) or change the deserialized type to an array or a type that implements a collection interface (e.g. ICollection, IList) like List<T> that can be deserialized from a JSON array. JsonArrayAttribute can also be added to the type to force it to deserialize from a JSON array.
I have tried changing my deserialization to:
dim result As cards = JsonConvert.DeserializeObject(Of List(Of cards))(json)
but to no available, something that was mentioned in another question, can someone point me in the right direction here please, I feel like I've been looking at it too long and need another pair of eyes.

Is there a way to extract part of a dataframe using pandas (or other library) in Python?

Hi I am new on Python and pandas. So please excuse my mistakes.
I want to know if it's possible to use a dataframe and sort it with pandas.
I'm still in class, but I'd like to advance in my projects.
I want / I'd like to extract part of the JSON file.
I want all the system_name =Métro de Paris to be in a JSON file.
Following are the current code I have (JSON and script).
Thanks in advance for your help!
import pandas as pd
# parse according to "system_name":"Métro de Paris"
data = pd.read_json(r'paris_lines_systems_and_modes.json')
[{"id":406,"name":"Ancien M 2nord","url_name":"406-ancien-m-2nord","color":"#000","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":405,"name":"Ancien M 2sud","url_name":"405-ancien-m-2sud","color":"#000","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":399,"name":"Ancien M 14","url_name":"399-ancien-m-14","color":"#000","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":460,"name":"Ancienne ligne A Nord-Sud","url_name":"460-ancienne-ligne-a-nord-sud","color":"#000","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":459,"name":"Ancienne ligne B Nord-Sud","url_name":"459-ancienne-ligne-b-nord-sud","color":"#000","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":411,"name":"Austerlitz-Auteuil","url_name":"411-austerlitz-auteuil","color":"#000","system_id":303,"system_name":"Navettes fluviales","transport_mode_id":0,"transport_mode_name":"default"},{"id":471,"name":"CDG Express","url_name":"471-cdgexpress","color":"#f0871f","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":7,"transport_mode_name":"people_mover"},{"id":465,"name":"CDGVal 1","url_name":"465-cdgval-1","color":"#0a3071","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":7,"transport_mode_name":"people_mover"},{"id":466,"name":"CDGVal 2","url_name":"466-cdgval-2","color":"#0a3071","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":7,"transport_mode_name":"people_mover"},{"id":410,"name":"Charenton-Auteuil","url_name":"410-charenton-auteuil","color":"#000","system_id":303,"system_name":"Navettes fluviales","transport_mode_id":0,"transport_mode_name":"default"},{"id":407,"name":"Funiculaire de Monmartre","url_name":"407-funiculaire","color":"#601e92","system_id":302,"system_name":"Funiculaire","transport_mode_id":0,"transport_mode_name":"default"},{"id":381,"name":"H","url_name":"381-h","color":"#7b4339","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":382,"name":"J","url_name":"382-j","color":"#cdcd00","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":383,"name":"K","url_name":"383-k","color":"#c7b300","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":384,"name":"L","url_name":"384-l","color":"#7584bc","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":472,"name":"Le Bus Direct 1","url_name":"472-le-bus-direct-1","color":"#b20066","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":8,"transport_mode_name":"bus"},{"id":473,"name":"Le Bus Direct 2","url_name":"473-le-bus-direct-2","color":"#7bbb58","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":8,"transport_mode_name":"bus"},{"id":474,"name":"Le Bus Direct 3","url_name":"474-le-bus-direct-3","color":"#0098d8","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":8,"transport_mode_name":"bus"},{"id":475,"name":"Le Bus Direct 4","url_name":"475-le-bus-direct-4","color":"#ff9b00","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":8,"transport_mode_name":"bus"},{"id":412,"name":" Le Louvre-Ablon","url_name":"412-le-louvre-ablon","color":"#000","system_id":303,"system_name":"Navettes fluviales","transport_mode_id":0,"transport_mode_name":"default"},{"id":643,"name":"Ligne Horizon","url_name":"643-ligne-horizon","color":"#0092e2","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":352,"name":"M 1","url_name":"352-ligne-1","color":"#ffcd00","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":353,"name":"M 2","url_name":"353-ligne-2","color":"#006db8","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":354,"name":"M 3","url_name":"354-ligne-3","color":"#9b993b","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":355,"name":"M 3bis","url_name":"355-ligne-3bis","color":"#88d3df","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":356,"name":"M 4","url_name":"356-ligne-4","color":"#ba4a9a","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":357,"name":"M 5","url_name":"357-ligne-5","color":"#f78e4b","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":358,"name":"M 6","url_name":"358-ligne-6","color":"#77c596","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":303,"name":"M 7","url_name":"303-ligne-7","color":"#f49eb2","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":359,"name":"M 7bis","url_name":"359-ligne-7bis","color":"#77c596","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":360,"name":"M 8","url_name":"360-ligne-8","color":"#c5a3cc","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":361,"name":"M 9","url_name":"361-ligne-9","color":"#cdc92b","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":362,"name":"M 10","url_name":"362-ligne-10","color":"#dfb03a","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":363,"name":"M 11","url_name":"363-ligne-11","color":"#8c6539","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":364,"name":"M 12","url_name":"364-ligne-12","color":"#008c59","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":365,"name":"M 13","url_name":"365-ligne-13","color":"#87d2df","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":366,"name":"M 14","url_name":"366-ligne-14","color":"#662c91","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":395,"name":"M 15","url_name":"395-m-15","color":"#a60f31","system_id":300,"system_name":"Grand Paris Express","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":396,"name":"M 16","url_name":"396-m-16","color":"#ec7cae","system_id":300,"system_name":"Grand Paris Express","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":397,"name":"M 17","url_name":"397-m-17","color":"#ec7cae","system_id":300,"system_name":"Grand Paris Express","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":398,"name":"M 18","url_name":"398-m-18","color":"#95bf32","system_id":300,"system_name":"Grand Paris Express","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":385,"name":"N","url_name":"385-n","color":"#00a092","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":476,"name":"Navette","url_name":"476-navette","color":"#000","system_id":83,"system_name":"Métro de Paris","transport_mode_id":4,"transport_mode_name":"heavy_rail"},{"id":394,"name":"Orlyval","url_name":"394-orlyval","color":"#7d7e7e","system_id":299,"system_name":"Dessertes aéroportuaires","transport_mode_id":7,"transport_mode_name":"people_mover"},{"id":386,"name":"P","url_name":"386-p","color":"#f0b600","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":387,"name":"R","url_name":"387-r","color":"#e4b4d1","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":376,"name":"RER A","url_name":"376-rer-a","color":"#ec1b2a","system_id":297,"system_name":"RER","transport_mode_id":3,"transport_mode_name":"commuter_rail"},{"id":377,"name":"RER B","url_name":"377-rer-b","color":"#4c90cc","system_id":297,"system_name":"RER","transport_mode_id":3,"transport_mode_name":"commuter_rail"},{"id":378,"name":"RER C","url_name":"378-rer-c","color":"#ffdc1d","system_id":297,"system_name":"RER","transport_mode_id":3,"transport_mode_name":"commuter_rail"},{"id":379,"name":"RER D","url_name":"379-rer-d","color":"#00ab66","system_id":297,"system_name":"RER","transport_mode_id":3,"transport_mode_name":"commuter_rail"},{"id":380,"name":"RER E","url_name":"380-rer-e","color":"#c67db4","system_id":297,"system_name":"RER","transport_mode_id":3,"transport_mode_name":"commuter_rail"},{"id":414,"name":"STCRP","url_name":"414-stcrp","color":"#000","system_id":303,"system_name":"Navettes fluviales","transport_mode_id":0,"transport_mode_name":"default"},{"id":367,"name":"T 1","url_name":"367-ligne-1","color":"#14609e","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":368,"name":"T 2","url_name":"368-ligne-2","color":"#ba498d","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":369,"name":"T 3a","url_name":"369-ligne-3a","color":"#ef7d2d","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":370,"name":"T 3b","url_name":"370-ligne-3b","color":"#009b41","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":371,"name":"T 4","url_name":"371-ligne-4","color":"#d9922b","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":372,"name":"T 5","url_name":"372-ligne-5","color":"#914e8e","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":373,"name":"T 6","url_name":"373-ligne-7","color":"#e95c36","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":375,"name":"T 7","url_name":"375-t-7","color":"#885f30","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":374,"name":"T 8","url_name":"374-t-8","color":"#8c8235","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":389,"name":"T 9","url_name":"389-t-9","color":"#009cb4","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":390,"name":"T 10","url_name":"390-t-10","color":"#8c8235","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":391,"name":"T 11","url_name":"391-t-11","color":"#df5f3d","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":506,"name":"T 11 express","url_name":"506-t-11-express","color":"#e2562e","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":392,"name":"T 12","url_name":"392-t-12","color":"#a02a3a","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":393,"name":"T 13","url_name":"393-t-13","color":"#885f30","system_id":296,"system_name":"Tramway d'Île-de-France","transport_mode_id":5,"transport_mode_name":"light_rail"},{"id":400,"name":"T Zen 1","url_name":"400-t-zen-1","color":"#c6cd01","system_id":301,"system_name":"TZen","transport_mode_id":6,"transport_mode_name":"brt"},{"id":401,"name":"T Zen 2","url_name":"401-t-zen-2","color":"#c6cd01","system_id":301,"system_name":"TZen","transport_mode_id":6,"transport_mode_name":"brt"},{"id":402,"name":"T Zen 3","url_name":"402-t-zen-3","color":"#c6cd01","system_id":301,"system_name":"TZen","transport_mode_id":6,"transport_mode_name":"brt"},{"id":403,"name":"T Zen 4","url_name":"403-t-zen-4","color":"#c6cd01","system_id":301,"system_name":"TZen","transport_mode_id":6,"transport_mode_name":"brt"},{"id":404,"name":"T Zen 5","url_name":"404-t-zen-5","color":"#c6cd01","system_id":301,"system_name":"TZen","transport_mode_id":6,"transport_mode_name":"brt"},{"id":409,"name":"Tuileries-Suresnes","url_name":"409-tuileries-suresnes","color":"#000","system_id":303,"system_name":"Navettes fluviales","transport_mode_id":0,"transport_mode_name":"default"},{"id":388,"name":"U","url_name":"388-u","color":"#d60058","system_id":298,"system_name":"Transilien","transport_mode_id":2,"transport_mode_name":"inter_city_rail"},{"id":408,"name":"Voguéo","url_name":"408-vogueo","color":"#99d420","system_id":303,"system_name":"Navettes fluviales","transport_mode_id":0,"transport_mode_name":"default"}]
So I managed to find how do extract part of the data you want to use / analyse.
You should simply call the column you are trying to extract (in my case system_name) and give its value (in my case "Métro de Paris"). Here The dataframe, data, will now have the data with value "Métro de Paris" form the column "system_name".
data = data[data.system_name == "Métro de Paris"]

Golang YAML with nested properties to JSON

I'm trying to parse some YAML from a file and convert each item to a flat JSON object.
I'm having some issues with datasets where there are multiple / nested objects.
I'm failing to unmarshal multiple items per file.
I'm unsure how to get nested properties such has item -> description -> en
When I run the following code I get the error.
cannot unmarshal !!map into []item.Item
I think I've got to look into sub-types within my structure, but I can't seem to get anything to work.
Any help would be appreciated.
Code:
import (
"gopkg.in/yaml.v2"
"io/ioutil"
)
func parseItemYaml() {
filePath := "./typeIDs.yaml"
yamlFile, err := ioutil.ReadFile(filePath)
if err != nil {
log.Printf("yamlFile.Get err #%v ", err)
}
itemData := map[int][]Item{}
err = yaml.Unmarshal(yamlFile, &itemData)
if err != nil {
panic(err)
}
// Then convert to JSON
// Eventual persistance
}
Item Structure:
type Item struct {
ID int
Name string `yaml:"name"`
Description string `yaml:"description"`
GroupID int `yaml:"groupID"`
IconID int `yaml:"iconID"`
GraphicID int `yaml:"graphicID"`
MarketGroupID int `yaml:"marketGroupID"`
Mass float64 `yaml:"mass"`
Published bool `yaml:"published"`
Volume float64 `yaml:"volume"`
}
Desired JSON format:
{
"ID":1563,
"Description":"Radiates an omnidirectional pulse from the ship that causes EM damage to surrounding vessels.",
"GroupID":72,
"IconID":112,
"GraphicID":72,
"MarketGroupID":382,
"Mass":10.0,
"Published":true,
"Volume":12.5
},
{...},
{...}
Data Subset:
1563:
description:
de: Ein ungerichteter Rundum-Impuls der vom Schiff ausgestrahlt wird und bei
umliegenden Schiffen EM-Schaden verursacht.
en: Radiates an omnidirectional pulse from the ship that causes EM damage
to surrounding vessels.
fr: Émet une impulsion omnidirectionelle qui part du vaisseau et provoque
des dégâts électromagnétiques à tous les vaisseaux situés à proximité.
ja: 全方位にパルスを放射し、周囲の艦にEMダメージを与える。
ru: Испускает сильный всенаправленный импульс, наносящий электромагнитный
ущерб всем объектам, окружающим корабль.
zh: 从舰船上发射一个全向脉冲,对周围船只造成电磁伤害。
graphicID: 2032
groupID: 72
iconID: 112
marketGroupID: 382
mass: 10.0
name:
de: Small EMP Smartbomb I
en: Small EMP Smartbomb I
fr: Petite bombe de proximité à IEM I
ja: 小型EMPスマートボムI
ru: Small EMP Smartbomb I
zh: 小型EMP立体炸弹 I
portionSize: 1
published: true
radius: 1000.0
volume: 12.5
1564:
basePrice: 650000.0
groupID: 152
iconID: 112
marketGroupID: 341
name:
de: Small EMP Smartbomb I Blueprint
en: Small EMP Smartbomb I Blueprint
fr: Plan de construction Petite bombe de proximité à IEM I
ja: 小型EMPスマートボムIブループリント
ru: Small EMP Smartbomb I Blueprint
zh: 小型EMP立体炸弹蓝图 I
portionSize: 1
published: true
volume: 0.01
1565:
description:
de: Ein ungerichteter Rundum-Impuls der vom Schiff ausgestrahlt wird und bei
umliegenden Schiffen EM-Schaden verursacht.
en: Radiates an omnidirectional pulse from the ship that causes EM damage
to surrounding vessels.
fr: Émet une impulsion omnidirectionelle qui part du vaisseau et provoque
des dégâts électromagnétiques à tous les vaisseaux situés à proximité.
ja: 全方位にパルスを放射し、周囲の艦にEMダメージを与える。
ru: Испускает сильный всенаправленный импульс, наносящий электромагнитный
ущерб всем объектам, окружающим корабль.
zh: 从舰船上发射一个全向脉冲,对周围船只造成电磁伤害。
graphicID: 2032
groupID: 72
iconID: 112
marketGroupID: 382
mass: 40.0
name:
de: Small EMP Smartbomb II
en: Small EMP Smartbomb II
fr: Petite bombe de proximité à IEM II
ja: 小型EMPスマートボムII
ru: Small EMP Smartbomb II
zh: 小型EMP立体炸弹 II
portionSize: 1
published: true
radius: 1000.0
volume: 5.0
Make structs e.g. like this:
type Language struct {
De string `yaml:de`
En string `yaml:en`
Fr string `yaml:fr`
Ja string `yaml:ja`
Ru string `yaml:ru`
Zh string `yaml:zh`
}
type Item struct {
Description Language `yaml:description`
GroupID int `yaml:"groupID"`
IconID int `yaml:"iconID"`
GraphicID int `yaml:"graphicID"`
MarketGroupID int `yaml:"marketGroupID"`
Mass float64 `yaml:"mass"`
Name Language `yaml:name`
Published bool `yaml:"published"`
Volume float64 `yaml:"volume"`
}
Also, as far as I see, it should be map[int]Item{} and not map[int][]Item{}.

How to select a specific record from a JSON source in YQL?

I'm trying to use YQL to get some data form a JSON source, but I'm getting NULL.
This is the JSON:
{
"categorias": [
{
"id": 0,
"categoryName": "Seguridad",
"description": "Protegete contra vulnerabilidades practicando correctamente la seguridad en el desarrollo de software",
"imageUrl": "http://domain/img_thmb/lock.png",
"tutorials": [
{
"id": 1,
"name": "Cross-Site request forgery (CSRF) para dummies",
"description": "Continuando con el tema de seguridad, vamos a analizar lo básico de un ataque CSRF para que puedan entender como funciona y poder protegerse del mismo.",
"tutorialUrl": "http://domain/post/6529205521/cross-site-request-forgery-csrf-para-dummies",
"imageUrl": "http://domain/lmsnt9akJ81qjthf4.jpg"
},
{
"id": 2,
"name": "Cross-site scripting (XSS) para dummies",
"description": "De acuerdo al OWASP el Cross-site scripting es una de las amenazas mas explotadas en el 2010.",
"tutorialUrl": "http://domain/post/6511050937/cross-site-scripting-xss-para-dummies",
"imageUrl": "http://domain/lmrfitzlmu1qjthf4.jpg"
}
]
},.....
This is the select statement I'm using in the YQL Console:
select * from json where url='http://domain/js/json.js' and id=1
I'm trying to get the "categorias ID" and I'm getting null, what am I doing wrong?
Im sorry, im new with YQL, im answering my own question, if there's some new too:
select categorias from json where url='http://domain/js/json.js' and categorias.id=1
Another option is to use the itemPath key to specify what should be considered an individual result.
select * from json
where itemPath="json.categorias"
and url="<url to json>"
and id=1;