I am geocoding using OpenRefine. I pulled data from OpenStreetMaps to my datasetstructure of data
I am adding a "column based on this column" for the coordinates.I want to check that the display_name contains "Rheinland-Pfalz" and if it does, I want to extract the latitude and longitude,i.e. pair.lat + ',' + pair.lon. I want to do this iteratively but I don't know how. I have tried the following:
if(display_name[0].contains("Rheinland-Pfalz"), with(value.parseJson()[0], pair, pair.lat + ',' + pair.lon),"nothing")
but I want to do this for each index [0] up to however many there are. I would appreciate if anyone could help.
Edit: Thanks for your answer b2m.
How would I extract the display_name corresponding to the coordinates that we get. I want the output to be display_name lat,lon for each match (i.e. contains "Rheinland-Pfalz", because I have a different column containing a piece of string that I want to match with the matches generated already.
For example, using b2m's code and incorporating the display_name in the output we get 2 matches:
Schaumburg, Balduinstein, Diez, Rhein-Lahn-Kreis, Rheinland-Pfalz, Deutschland 50.33948155,7.9784308849342604
Schaumburg, Horhausen, Flammersfeld, Landkreis Altenkirchen, Rheinland-Pfalz, Deutschland 52.622319,14.5865283
For each row, I have another string in a different column. Here the entry is "Rhein-Lahn-Kreis". I want to filter the two matches above to only keep those containing my string in the other column. In this case "Rhein-Lahn-Kreis" but the other column entry is different for each row. I hope this is clear and I would greatly appreciate any help
Assuming we have the following json data
[
{"display_name": "BW", "lat": 0, "lon": 1},
{"display_name": "NRW 1", "lat": 2, "long": 3},
{"display_name": "NRW 2", "lat": 4, "lon": 5}
]
You can extract the combined elements lat and long with forEach and filter using the following GREL expression e.g. in the Add column based on this column dialog.
forEach(
filter(
value.parseJson(), geodata, geodata.display_name.contains("NRW")
), el, el.lat + "," + el.lon)
.join(";")
This will result in a new field with the value 2,3;4,5.
You can then split the new multi valued field on the semicolon ";" to obtain separated values (2,3 and 4,5).
Another approach would be to split the JSON Array elements into separate rows, avoiding the forEach and filter functions.
Related
Lets say I have these arrayed JSON values
[{operation_id: 2, operation_name: FAITHFUL BELIEVERS},
{operation_id: 3, operation_name: SAMPLE OP},
{operation_id: 4, operation_name: SAMPLE OP 2}]
Now I will select the operation name 'SAMPLE OP' but I want to display the value of its operation_id. How would I do that?
Your JSON is a list of maps, so use where on the list to filter it by your predicate. Better still, use firstWhere as we assume there's just one match.
The match function returns true if the operation name member of the map matches.
firstWhere returns the first matching map, and you want the operation id member of that map.
final id = list
.firstWhere((m) => m['operation_name'] == 'SAMPLE OP')['operation_id'];
Given the table:
C1 C2 C3
----------------
1 'v1' 1.1
2 'v2' 2.2
3 'v3' 3.3
Is there any "easy" way to return JSON in this format:
{
"columns": [ "C1", "C2", "C3" ],
"rows": [
[ 1, "v1", 1.1 ],
[ 2, "v2", 2.2 ],
[ 3, "v3", 3.3 ]
]
}
To generate an array with single values from a table there is a neat trick like this:
SELECT JSON_QUERY(REPLACE(REPLACE(
(
SELECT id
FROM table a
WHERE pk in (1,2)
FOR JSON PATH
), '{"id":',''),'}','')) 'ids'
Which generates
"ids": [1,2]
But to construct the nested array above the replacing gets really tedious, anyone know a good way to achieve this?
Well, you ask for an easy way but the following will not be easy :-)
The tricky part is to know which values need to be qouted and which can remain naked.
This needs generic type-analysis to find, which values are strings.
The only way I know to get on meta data (besides building dynamic sql using meta views like INFORMATIONSCHEMA.COLUMNS) is XML together with an AUTO-schema.
This XML is very near to your needs actually. There is a list of columns at the beginning, followed by a list of rows. But it is not JSON of course...
Try this out:
--This is a mockup table with the values you provided.
DECLARE #mockup TABLE(C1 INT,C2 VARCHAR(100),C3 DECIMAL(4,2));
INSERT INTO #mockup VALUES
(1,'v1',1.1)
,(2,'v2',2.2)
,(3,'v3',3.3);
--Now we create an XML out of this
DECLARE #xml XML =
(
SELECT *
FROM #mockup t
FOR XML RAW,XMLSCHEMA,TYPE
);
--Check the XML's content with SELECT #xml to see how it is looking internally
--Now the real query can start:
SELECT '{"columns":[' +
STUFF(#xml.query('declare namespace xsd="http://www.w3.org/2001/XMLSchema";
for $col in /xsd:schema/xsd:element//xsd:attribute
return
<x>,{concat("""",xs:string($col/#name),"""")}</x>
').value('.','nvarchar(max)'),1,1,'') +
'],"rows":[' +
STUFF(
(
SELECT
',[' + STUFF(b.query(' declare namespace xsd="http://www.w3.org/2001/XMLSchema";
for $attr in ./#*
return
<x>,{if(/xsd:schema/xsd:element//xsd:attribute[#name=local-name($attr)]//xsd:restriction/#base="sqltypes:varchar") then
concat("""",$attr,"""")
else
xs:string($attr)
}
</x>
').value('.','nvarchar(max)'),1,1,'') + ']'
FROM #xml.nodes('/*:row') B(b)
FOR XML PATH(''),TYPE
).value('.','nvarchar(max)'),1,1,'') +
']}';
The result
{"columns":["C1","C2","C3"],"rows":[[3,"v3",3.30],[1,"v1",1.10],[2,"v2",2.20]]}
Some explanation:
The first part will use XQuery to find all columns (xsd:attribute within XML-schema) and create the array of column names.
The second part will againt use XQuery in order to run through all rows and write their column values in a concatenated string. Each value can refer to its type within the schema. Whenever this type is sqltypes:varchar the value will be quoted. All other values remain naked.
This will not solve each and any case generically...
To be honest, this was more for my own curiosity :-) Wanted to find out, how one can solve this.
Quite probably the best answer is: Use another tool. SQL-Server is not the best choice here ;-)
I have a JSON field in mysql for tags like so
[
{"id": 138001, "tag": "Travel"},
{"id": 138002, "tag": "Holidays"}
]
and I want to retrieve it as 'Travel Holidays'
I can get the values in the a json array format:
["Travel", "Holidays"]
with the current statement,
SELECT JSON_EXTRACT(post_tags, '$[*].tag') AS tags FROM posts WHERE post_id = 646745;
but I'm not sure if there is a way to flatten the array into a single string separated by spaces, is it possible?
My reason for doing this, is that sphinx search indexes the tags field also and you can get the id values in the index which isn't desired at all
Well you could just let sphinx index ["Travel", "Holidays"] directly.
As per charset_table rules, the [, " and , will just be 'ignored' (more technically treated as separators and collapsed away)
Also not familiar with it, but maybe JSON_UNQUOTE could work? Don't know if it can collapse arrays.
I am trying to extract data from some JSON with JQ - I have already got it down to the last level of data that I need to extract from, but I am completely stumped as to how to proceed with how this part of the data is formatted.
An example would be:
{
"values": [
[
1483633677,
42
]
],
"columns": [
"time",
"count_value"
],
"name": "response_time_error"
}
I would want to extract just the value for a certain column (e.g. count_value) and I can extract it by using [-1] in this specific case, but I want to select the column by its name in case they change in the future.
If you're only extracting a single value and the arrays will always correspond with eachother, you could find the index in the columns array then use that index into the values array.
It seems like values is an array of rows with those values. Assuming you want to output the values of all rows with the selected column:
$ jq --arg col 'count_value' '.values[][.columns | index($col)]' input.json
If the specified column name does not exist in .columns, then Jeff's filter will fail with a rather obscure error message. It might therefore be preferable to check whether the column name is found. Here is an illustration of how to do so:
jq --arg col count_value '
(.columns | index($col)) as $ix
| if $ix then .values[][$ix] else empty end' input.json
If you want an informative error message to be printed, then replace empty with something like:
error("specified column name, \($col), not found")
I have a JSON result I am trying to work with in AppleScript, but because the top level items are "unnamed" I can only access them by piping the item reference, which in this case is a number. As a result, I can't iterate through it, it has to be hard coded (scroll down to the last code sample to see what I mean)
For example, this is the JSON I'm looking at:
{
"1": {
"name": "Tri 1"
},
"2": {
"name": "Tri 2"
},
"3": {
"name": "Tri 3"
},
"4": {
"name": "Orb Dave"
},
"5": {
"name": "Orb Fah"
}
}
With the help of JSON Helper I get the JSON to a more usable format (for AppleScript).
{|3|:{|name|:"Tri 3"}, |1|:{|name|:"Tri 1"}, |4|:{|name|:"Orb Dave"}, |2|:{|name|:"Tri 2"}, |5|:{|name|:"Orb Fah"}}
I can then use this code to get a list of "lights" the objects in question:
set lights to (every item in theReturn) as list
repeat with n from 1 to count of lights
set light to item n of lights
log n & light
end repeat
From that, I get:
(*1, Tri 3*)
(*2, Tri 1*)
(*3, Orb Dave*)
(*4, Tri 2*)
(*5, Orb Fah*)
You may notice the result is not in the desired order. The index is the index within the list of lights. It's not the number that appears at the top of the object. If you look to the top two pre-formated areas, you'll see the items 1,2 and 3 are Tri 1, Tri 2, and Tri 3. It is correct that Tri 3 comes first, Tri 1 second, and an Orb is third.
What I need to do is find a way to be able to iterate through the JSON in any order (sorted or not) and be able to line up "1" with "Tri 1", "3" with "Tri 3" and "5" with "Orb Fah". But I can't find ANY way to interact with the returned JSON that lets me reference the third light and return it's name. The ONLY way I can seem to be able to do it is to hard code the light indexes, such that:
log |name| of |1| of theReturn
log |name| of |2| of theReturn
log |name| of |3| of theReturn
log |name| of |4| of theReturn
log |name| of |5| of theReturn
which gives me the correct light with the correct name:
(*Tri 1*)
(*Tri 2*)
(*Tri 3*)
(*Orb Dave*)
(*Orb Fah*)
I'm thinking the problem is arising because the light ID doesn't have a descriptor or sorts. That I can't change, but I need to iterate through them programatically. Hard coding them as above is not acceptable.
Any help would be appreciated
You are dealing with a list of records here, not a list of lists. Records are key/value pairs. They do not have indexes like a list. That makes it easy if you know the keys because you just ask for the one you want. And your records have records inside them so you have 2 layers of records. Therefore if you want the value of the |name| record corresponding to |3| record then ask for it as you've discovered...
set jsonRecord to {|3|:{|name|:"Tri 3"}, |1|:{|name|:"Tri 1"}, |4|:{|name|:"Orb Dave"}, |2|:{|name|:"Tri 2"}, |5|:{|name|:"Orb Fah"}}
set record3name to |name| of |3| of jsonRecord
The downside of records in applescript is that there is no command to find the record keys. Other programming languages give you the tools to find the keys (like objective-c) but applescript does not. You have to know them ahead of time and use them as I showed.
If you don't know the keys ahead of time then you can either use JSON Helper to give you the results in a different form or use a different programming language (python, ruby, etc) to extract the information from the records.
One other option you have is to just use the json text itself without using JSON Helper. For example, if you have the json as text then you can extract the information using standard applescript commands for text objects. Your json text has the information you want on the 3rd line, the 6th, 9th etc. You could use that to your advantage and do something like this...
set jsonText to "{
\"1\": {
\"name\": \"Tri 1\"
},
\"2\": {
\"name\": \"Tri 2\"
},
\"3\": {
\"name\": \"Tri 3\"
},
\"4\": {
\"name\": \"Orb Dave\"
},
\"5\": {
\"name\": \"Orb Fah\"
}
}"
set jsonList to paragraphs of jsonText
set namesList to {}
set AppleScript's text item delimiters to ": \""
repeat with i from 3 to count of jsonList by 3
set theseItems to text items of (item i of jsonList)
set end of namesList to text 1 through -2 of (item 2 of theseItems)
end repeat
set AppleScript's text item delimiters to ""
return namesList
For each index, loop through all the items in the list looking for the one whose name matches the index:
tell application "System Events"
-- Convert the JSON file to a property list using plutil.
do shell script "plutil -convert xml1 /Users/mxn/Desktop/tri.json -o /Users/mxn/Desktop/tri.plist"
-- Read in the plist
set theItems to every property list item of property list file "/Users/mxn/Desktop/tri.plist"
set theLights to {}
-- Iterate once per item in the plist.
repeat with i from 1 to count of theItems
set theName to i as text
-- Find the item whose name is the current index.
repeat with theItem in theItems
if theItem's name is theName then
-- We found it, so add it to the results.
set theValue to theItem's value
copy {i, theValue's |name|} to the end of theLights
-- Move on to the next index.
exit repeat
end if
end repeat
end repeat
return theLights
end tell
Result:
{{1, "Tri 1"}, {2, "Tri 2"}, {3, "Tri 3"}, {4, "Orb Dave"}, {5, "Orb Fah"}}
Ideally, instead of the nested loop, we’d be able to say something like this:
set theName to i as text
set theItem to (the first item in theItems whose name is theName)
But unfortunately that produces an error.
This solution also demonstrates an alternative to JSON Helper: you can convert the JSON file to a property list using the handy plutil command line tool and use System Events' built-in support for property lists.