I am still taking baby steps with Yahoo Pipes and struggling with what I believe should be a simple task.
I have a table on a page that is being updated in realtime (every 1-2 minutes).
I want to extract the rows, push into a Pipe, and then spit out a JSON in the following format:
"sites": [
{
"Site": "210001-Singleton",
"LastSampleTime": "29/04/2014 11:51:00",
"RiverLevel": "0.744",
"FlowRate": "501.6",
"FlowRate": "0.744",
"Rainfall": "",
"WaterTemp": "",
"Conductivity": ""
},
etc.
I think I am right in thinking that once I have pulled the relative table components with an XPath fetcher, I would make use of a Loop with an Item Builder contained within it to spit out the data into the above format. However, I am struggling with trying to pull in the simple table.
Here is a simplified version of my yahoo pipe.
I have tried multiple variations of the XPath string to try and get just the rows I need.
From inspecting the table with firebug, I know that the TRs I want, all seem to share the same height of tr style="height:18px"
However, not sure if this is the best way to extract them.
Can someone assist in providing some pointers on how to pull the table into my desired format? Not too sure where I am going wrong with Xpath
Import.io can do what you want. Even though the HTML on that site is a bit messy, you can still use a custom xpath override within the tool.
I built the first row of data for you, so all you need to do is go in and edit the existing extractor adding in more columns using the following extractor as a start point https://import.io/data/set/?mode=loadSource&source=f867a123-091e-4596-bbea-871df2d5ceb7
Just open it up, edit the extractor and add the cols you need. Here is the xPath code I used:
/html/body/table/tbody/tr[7]/td[5]
Row 7 in the table is the first row with data, and td[2] is the first cell in. Just increase the number in the tr[x] to hit the next row.
Once you have the data structured, hit integrate, and follow the instructions. Use the import.io support to help too, thats what they are there for.
If the table will be expanded with more rows, you may want to change the xPath to work off of the values of the child element of the tr
Disclaimer: I work at import.io, other tools exist.
Related
I have a JSON file with five sub-levels that I need to parse with PDI. I am wondering if I can do this with one JSON input step or do I need to process each level separately. Attached is the screenshot of what I have so far.
Will PDI process all the lower-level data before moving to the next parent level or do I need to parse the top level and then pass that on to another JSON.input step?
i tried working on version 6.2 with 3 levels and had to use multiple json input.for quicker execution
passed everything as data stream
in my example, i was working with order headers and order lines so separated them functionally using one json input for header to populate header details and other for order lines. may be you can use the same approach
if you are using version 8.2 i think this should work for you
https://help.pentaho.com/Documentation/8.3/Products/JSON_Input
I have this solution that helps me creating a Wizard to fill some data and turn into JSON, the problem now is that I have to receive a xlsx and turn specific data from it into JSON, not all the data but only the ones I want which are documented in the last link.
In this link: https://stackblitz.com/edit/xlsx-to-json I can access the excel data and turn into object (when I print document.getElementById('output').innerHTML = JSON.parse(dataString); it shows [object Object])
I want to implement this solution and automatically get the specified fields in the config.ts but can't get to work. For now, I have these in my HTML and app-component.ts
https://stackblitz.com/edit/angular-xbsxd9 (It's probably not compiling but it's to show the code only)
It wasn't quite clear what you were asking, but based on the assumption that what you are trying to do is:
Given the data in the spreadsheet that is uploaded
Use a config that holds the list of column names you want returned in the JSON when the user clicks to download
based on this, I've created a fork of your sample here -> Forked Stackbliz
what I've done is:
use the map operator on the array returned from the sheet_to_json method
Within the map, the process is looping through each key of the record (each key being a column in this case).
If a column in the row is defined in the propertymap file (config), then return it.
This approach strips out all columns you don't care about up front. so that by the time the user clicks to download the file, only the columns you want are returned. If you need to maintain the original columns, then you can move this logic somewhere more convenient for you.
I also augmented the property map a little to give you more granular control over how to format the data in the returned JSON. i.e. don't treat numbers as strings in the final output. you can use this as a template if it suites your needs for any additional formatting.
hope it helps.
I have a slew of JSON files I'm getting dumps of, with data from the day/period it was pulled. Most of the JSON files I'm dealing with are a lot larger than this, but I figured a smaller one would be easier to work with.
{"playlists":[{"uri":"spotify:user:11130196075:playlist:1Ov4b3NkyzIMwfY9E8ixpE","listeners":366,"streams":386,"dateAdded":"2016-02-24","newListeners":327,"title":"#Covers","owner":"Saga Prommeedet"},{"uri":"spotify:user:mickeyrose30:playlist:2Ov4b3NkyzIMwfY9E8ixpE","listeners":229,"streams":263,"dateAdded":"removed","newListeners":154,"title":"bestcovers2016","owner":"Mickey Rose"}],"top":2,"total":53820}
What I'm essentially trying to do is add a date attribute to each line of data, so that when I combine multiple JSON files to put through an analytical tool, the right row of data is associated with the correct date. My first thought was to write it as such:
{"playlists":[{"uri":"spotify:user:11130196075:playlist:1Ov4b3NkyzIMwfY9E8ixpE","listeners":366,"streams":386,"dateAdded":"2016-02-24","newListeners":327,"title":"#Covers","owner":"Saga Prommeedet"},{"uri":"spotify:user:mickeyrose30:playlist:2Ov4b3NkyzIMwfY9E8ixpE","listeners":229,"streams":263,"dateAdded":"removed","newListeners":154,"title":"bestcovers2016","owner":"Mickey Rose"}],"top":2,"total":53820,"date":072617}
since the "top" and "total" attributes are showing up on each row of data (with the associated values also showing up on each row) when I put it through an analytical tool like Tableau.
Also, have been editing and saving files through Brackets, and testing things through this converter (https://konklone.io/json/)
In javascript language
var m = JSON.parse(json_string);
m["date"]="20170804";
JSON.stringify(m);
This will work for you, very simple,
I'm working on some Python code for my local billiard hall and I'm running into problems with JSON encoding. When I dump my data into a file I obviously get all the data in a single line. However, I want my data to be dumped into the file following the format that I want. For example (Had to do picture to get point across),
My custom JSON format
. I've looked up questions on custom JSONEncoders but it seems they all have to do with datatypes that aren't JSON serializable. I never found a solution for my specific need which is having everything laid out in the manner that I want. Basically, I want all of the list elements to on a separate row but all of the dict items to be in the same row. Do I need to write my own custom encoder or is there some other approach I need to take? Thanks!
I have created a BC WebApp using the BC Open Platform API's for the back end and everything appears to work fine including rendering the list of items in a sort order of one of my custom fields.
Here is an example of what works on the back end.
var items = new BCAPI.Models.WebApp.ItemCollection(WEBAPP_NAME);
items.fetch({
order: "MyCustomField",
skip: 0,
limit: 1000,
success: onWebpAppListFetch,
error: onAPIError
});
How do I render this list on the front end sorted by one of my custom fields? Here is an example of what I am trying to use on the front end, but it does not order or sort this way.
{module_webapps order="MyCustomField" render="collection" template="/_System/apps/cms-sports-club-manager/club-rooms/layouts/club_rooms_collection.tpl" id="cms-club-roomsx" filter="all"}
Is there something that I am overlooking, or do I need to approach this in my own manual way? Perhaps I could render the list into an array, sort the array and then iterate through that to render the front end listing? The template file uses Liquid to iterate through the collection and render the HTML. Can I define an array variable, fill the array, sort the array and iterate through the array again in that same template file?
Another possibility perhaps is to output all webapp items into a JSON file each time a user creates/edits an item (from the back end), and then use the {module_json} feature on the front end to read that JSON file which "should" allow me to sort it.
Any advice on what to do (as well as what NOT to do) would be appreciated.
I have found one possible answer, but I am not sure it is the best way to do this.
I have implemented a script which performs the following steps:
Create an array holding all items (unsorted at this point)
Sort the array using any custom field
Iterate through the sorted array and document.write() each one
I have tested this a fair bit and it seems to do exactly what I want, however I have not marked this as the correct answer just yet as I would still like to find a "better" way than needing to resort to the above manual work.