This YQL statement (execute in the YQL console) picks out the name and year of a film from the TMDB website.
select content from html where url="http://www.themoviedb.org/movie/27205" and xpath='//h3[#id="year"]|//h2[#id="title"]/a'
the results come back like this:
"results": {
"a": "Inception",
"h3": "(2010)"
}
Is there any easy way to have label the results as 'name' and 'year' rather than the html elements they were grabbed from?
Cheers!
Currently there is no easy way to quickly alias the returned labels. The best way at the moment is to create a custom data table which makes your query to TMDB and transforms the result (with Javascript, in an <execute> block) to whatever you want it to be.
For example, I created a quick custom table for you which returns the results with your labels of choice (name and year). It also removes the parentheses from around the year. To give it a test run, use:
use "store://github.com/tmdb-jp" as tmdb;
select * from tmdb where movieid="27205"
With the resulting JSON having the following structure (within the usual YQL response):
"movie": {
"title": "Inception",
"year": "2010"
}
If you want to have a go at creating a data table yourself, or just see what's involved, then the source is on my github. Also (it might be useful) you can query for multiple movies at once:
use "store://github.com/tmdb-jp" as tmdb;
select * from tmdb where movieid in ("27205","9802")
P.S. The store:// URLs just mean that the data table is being stored in Yahoo!'s "cloud" for speed and reliability. You can of course use a normal http:// URL (e.g. by github one) instead.
Related
I am trying to use Zapier to add an email to a SendFox mailing list when a new user gets added to a specified Firestore path. It's asking me for a structured query to find this data.
I am using the one that it's suggesting, but new users aren't being added correctly. My concern is that the structured query isn't set up correctly.
My data is structured as follows:
- Customers (Collection)
- [User ID] (Document)
- EMAIL
- Other Info
- Other Info...
Whenever a new UserID document is added, I'm trying to access the email field in Zapier.
This is the current structured query:
"orderBy": [{
"field": {
"fieldPath": "email"
},
"direction": "DESCENDING"
}]
What is the correct structured query (json) to access the document I'm looking for?
To clarify, it's the latest document in the "customers" collection with the field "email".
I'm not looking fo any javascript code to get this data, merely the correct json structure for this query.
The query you have right now returns all documents from the customers collection in descending order of their email address, so with the zs first. That seems to be unlikely what you want.
Firestore has no built in concept of the most recent document, so what you'll want to do is:
Add a timestamp field to each document that you set to the current time (preferably a server-side timestamp, but client-side will probably work too).
Sort the query on descending values of that new timestamp field.
As part of a tool I am creating for my team I am connecting to an internal web service via PowerQuery.
The web service returns nested JSON, and I have trouble parsing the JSON data to the format I am looking for. Specifically, I have a problem with extracting the content of records in a column to a comma separated list.
The data
As you can see, the data contains details related to a specific "race" (race_id). What I want to focus on is the information in the driver_codes which is a List of Records. The amount of records varies from 0 to 4 and each record is structured as id: 50000 (50000 could be any 5 digit number). So it could be:
id: 10000
id: 20000
id: 30000
As requested, an example snippet of the raw JSON:
<race>
<race_id>ABC123445</race_id>
<begin_time>2018-03-23T00:00:00Z</begin_time>
<vehicle_id>gokart_11</vehicle_id>
<driver_code>
<id>90200</id>
</driver_code>
<driver_code>
<id>90500</id>
</driver_code>
</race>
I want it to be structured as:
10000,20000,30000
The problem
When I choose "Extract values" on the column with the list, then I get the following message:
Expression.Error: We cannot convert a value of type Record to type
Text.
If I instead choose "Expand to new rows", then duplicate rows are created for each unique driver code. I now have several rows per unique race_id, but what I wanted was one row per unique race_id and a concatenated list of driver codes.
What I have tried
I have tried grouping the data by the race_id, but the operations allowed when grouping data do not include concatenating rows.
I have also tried unpivoting the column, but that leaves me with the same problem: I still get multiple rows.
I have googled (and Stack Overflowed) this issue extensively without luck. It might be that I am using the wrong keywords, however, so I apologize if a duplicate exists.
UPDATE: What I have tried based on the answers so far
I tried Alexis Olson's excellent and very detailed method, but I end up with the following error:
Expression.Error: We cannot convert the value "id" to type Number. Details:
Value=id
Type=Type
The error comes from using either of these lines of M code (one with a List.Transform and one without):
= Table.Group(#"Renamed Columns", {"race_id", "begin_time", "vehicle_id"},
{{"DriverCodes", each Text.Combine([driver_code][id], ","), type text}})
= Table.Group(#"Renamed Columns", {"race_id", "begin_time", "vehicle_id"},
{{"DriverCodes", each Text.Combine(List.Transform([driver_code][id], each Number.ToText(_)), ","), type text}})
NB: if I do not write [driver_code][id] but only [id] then I get another error saying that column [id] does not exist.
Here's the JSON equivalent to the XML example you gave:
{"race": {
"race_id": "ABC123445",
"begin_time": "2018-03-23T00:00:00Z",
"vehicle_id": "gokart_11",
"driver_code": [
{ "id": "90200" },
{ "id": "90500" }
]}}
If you load this into the query editor, convert it to a table, and expand out the Value record, you'll have a table that looks like this:
At this point, choose Expand to New Rows, and then expand the id column so that your table looks like this:
At this point, you can apply the trick #mccard suggested. Group by the first columns and aggregate over the last using, say, max.
This last step produces M code like this:
= Table.Group(#"Expanded driver_code1",
{"Name", "race_id", "begin_time", "vehicle_id"},
{{"id", each List.Max([id]), type text}})
Instead of this, you want to replace List.Max with Text.Combine as follows:
= Table.Group(#"Changed Type",
{"Name", "race_id", "begin_time", "vehicle_id"},
{{"id", each Text.Combine([id], ","), type text}})
Note that if your id column is not in the text format, then this will throw an error. To fix this, insert a step before you group rows using Transform Tab > Data Type: Text to convert the type. Another options is to use List.Transform inside your Text.Combine like this:
Text.Combine(List.Transform([id], each Number.ToText(_)), ",")
Either way, you should end up with this:
An approach would be to use the Advanced Editor and change the operation done when grouping the data directly there in the code.
First, create the grouping using one of the operations available in the menu. For instance, create a column"Sum" using the Sum operation. It will give an error, but we should get the starting code to work on.
Then, open the Advanced Editor and find the code corresponding to the operation. It should be something like:
{{"Sum", each List.Sum([driver_codes]), type text}}
Change it to:
{{"driver_codes", each Text.Combine([driver_codes], ","), type text}}
I know how to parse json cells in Open refine, but this one is too tricky for me.
I've used an API to extract the calendar of 4730 AirBNB's rooms, identified by their IDs.
Here is an example of one Json file : https://fr.airbnb.com/api/v2/calendar_months?key=d306zoyjsyarp7ifhu67rjxn52tv0t20¤cy=EUR&locale=fr&listing_id=4212133&month=11&year=2016&count=12&_format=with_conditions
For each ID and each day of the year from now until november 2017, i would like to extract the availability of this rooms (true or false) and its price at this day.
I can't figure out how to parse out these informations. I guess that it implies a series of nested forEach, but i can't find the right way to do this with Open Refine.
I've tried, of course,
forEach(value.parseJson().calendar_months, e, e.days)
The result is an array of arrays of dictionnaries that disrupts me.
Any help would be appreciate. If the operation is too difficult in Open Refine, a solution with R (or Python) would also be fine for me.
Rather than just creating your Project as text, and working with GREL to parse out...
The best way is just select the JSON record part that you want to work with using our visual importer wizard for JSON files and XML files (you can even use a URL pointing to a JSON file as in your example). (A video tutorial shows how here: https://www.youtube.com/watch?v=vUxdB-nl0Bw )
Select the JSON part that contains your records that you want to parse and work with (this can be any repeating part, just select one of them and OpenRefine will extract all the rest)
Limit the amount of data rows that you want to load in during creation, or leave default of all rows.
Click Create Project and now your in Rows mode. However if you think that Records mode might be better suited for context, just import the project again as JSON and then select the next outside area of the content, perhaps a larger array that contains a key field, etc. In the example, the key field would probably be the Date, and why I highlight the whole record for a given date. This way OpenRefine will have Keys for each record and Records mode lets you work with them better than Row mode.
Feel free to take this example and make it better and even more helpful for all , add it to our Wiki section on How to Use
I think you are on the right track. The output of:
forEach(value.parseJson().calendar_months, e, e.days)
is hard to read because OpenRefine and JSON both use square brackets to indicate arrays. What you are getting from this expression is an OR array containing twelve items (one for each month of the year). The items in the OR array are JSON - each one an array of days in the month.
To keep the steps manageable I'd suggest tackling it like this:
First use
forEach(value.parseJson().calendar_months,m,m.days).join("|")
You have to use 'join' because OR can't store OR arrays directly in a cell - it has to be a string.
Then use "Edit Cells->Split multi-valued cells" - this will get you 12 rows per ID, each containing a JSON expression. Now for each ID you have 12 rows in OR
Then use:
forEach(value.parseJson(),d,d).join("|")
This splits the JSON down into the individual days
Then use "Edit Cells->Split multi-valued cells" again to split the details for each day into its own cell.
Using the JSON from example URL above - this gives me 441 rows for the single ID - each contains the JSON describing the availability & price for a single day. At this point you can use the 'fill down' function on the ID column to fill in the ID for each of the rows.
You've now got some pretty easy JSON in each cell - so you can extract availability using
value.parseJson().available
etc.
I am extremely new to Ruby on Rails, having only a couple days of experience. So far, I have created a new app, and loaded data into the database called name which is comprised of date:string, value:decimal, and unique_id:integer. So now, I can go to "(my local port)/name" and view the table successfully.
What I would like to do is this:
In a new html page, have a SIMILE Timeplot (http://www.simile-widgets.org/timeplot/) with an HTML drop-down list below it in order to select a unique_id and another drop-down box to select a year.
From there, I would like to search through the database and display all of the data on the Timeplot that matches the unique_id and that is in the specified year.
I believe I must make an HTTP GET request for a date_to, date_from, and unique_id, but I do not know how to implement this (admittedly I have been searching the web for ages, but could not figure out the solution).
Any help would be greatly appreciated! Thank you.
Edit: Even just advice on what component to tackle first
First you need to create a route for your search such as this:
match "name/search" to: "name#search" as: "name_search", via: :get
Then if you are using AJAX, and using jQuery you make an HTTP request like this:
$.get("/name/search", {
unique_id: <your_unique_id>,
date_from: <your_date_from>,
date_to: <your_date_to> },
function(result) {
// You do whatever you want with the result here
}
}
P.S:
The Javascript code might not be 100% correct, since I rarely use it.
I have a view called 'walking' which I want to query:
http://site/activity.nsf/walking?searchview&query=FIELD%20Gradient%20CONTAINS%20gradienteasy.gif
This returns the results in an HTML table. What I would like to do is have the results formatted as JSON which I will then use client-side. Is this possible?
I know you can get JSON returned from a straight view by doing this:
http://site/activity.nsf/walking?readviewentries&outputformat=json
Scott Good and I have done several sessions at a variety of conferences on generating and consuming JSON from traditional Domino applications (not using XPages). The most recent was the "JMP303 JSON in client- and server-side code Master Class" we gave at Lotusphere 2011. Link to the presentation materials and slides are: here
/Newbs
You will have to create a view that is marked with the "Treat view contents as HTML" property and set up a column formula that generates the JSON syntax that you want'.
There is a nice post on OpenNTF with the code to create a very generic view which returns JSON for the documents that match the View's selection formula:
http://openntf.org/XSnippets.nsf/snippet.xsp?id=use-transform-to-build-json-and-consume-the-output-in-an-xagent
That sample uses an "XAgent" (Xpage with no UI) to set the content-type header, etc. But you could probably do the same thing using a $$ViewTemplate form, if needed.