It seems that Yahoo pipes are represented using JSON. I want to download these JSON objects for some research purpose. Usually a Yahoo pipe is rendered in a browser editor thru a url like this: http://pipes.yahoo.com/pipes/pipe.edit?_id=XgRo96h13BGtJWvS8SvLAg, but you can't get the corresponding JSON object to this Yahoo pipe. Does anyone know how to get JSON objects representing Yahoo pipes and store them in any persistent form?
It is possible to get hold of a JSON description of a Yahoo Pipe using a URL of the form:
http://pipes.yahoo.com/pipes/pipe.info?_out=json&_id=PIPE_ID
The pipe2py python library demonstrates how to grab the JSON description of a pipe and "compile" it to a Python equivalent that can be run on your own server.
The post Exporting Yahoo Pipe Definitions, Compiling Them to Python, and Running Them in Scraperwiki describes how you can use pipe2py in the Scraperwiki environment to compile and execute pipes on Scraperwiki using pipe definitions imported directly from Yahoo Pipes, or exported from Yahoo Pipes and then stored locally in a Scraperwiki database table.
When I load that page in a browser I can see that it makes an ajax request for:
http://pipes.yahoo.com/pipes/ajax.pipe.load?id=XgRo96h13BGtJWvS8SvLAg&_out=json&modinfo=true&rnd=7560&.crumb=MjvGjpzhPLl
That's your object but I'm not sure if I'm answering your question of how to "get it". If you need to get it through a program you would need a script that loges into pipes and extracts that url.
A quick way, while not automated, is to use an HTTP analyzer. Here's a process for getting the object using HttpFox (I use v0.8.9) for Firefox. With the analyzer running, load the edit page for a pipe, like the one you linked:
http://pipes.yahoo.com/pipes/pipe.edit?_id=XgRo96h13BGtJWvS8SvLAg
Look at the request with a URL that starts with:
http://pipes.yahoo.com/pipes/ajax.pipe.load?id=....
Next, explore the content of the request (there's a 'Content' tab in HttpFox). That's the JSON object representing the pipe structure.
Use pipe.run?[your pipe id here]&_render=json as opposed to pipe.edit
So in your case to get the json it would be - http://pipes.yahoo.com/pipes/pipe.run?_id=XgRo96h13BGtJWvS8SvLAg&_render=json
I guess how you implement the client is dependent on what you like writing in/what other functionality you need.
You could also do it the other way around and use the web service service module to post the data to a script that can extract the json and persist it to a database. You could check out json.org.
Related
In my karate tests i need to write response id's to txt files (or any other file format such as JSON), was wondering if it has any capability to do this, I haven't seen otherwise in the documentation. In the case of no, is there a simple JavaScript function to do so?
Try the karate.write(value, filename) API but we don't encourage it. Also the file will be written only to the current "build" directory which will be target for Maven projects / stand-alone JAR.
value can be any data-type, and Karate will write the bytes (or plain-text) out. There is no built-in support for any other format.
Here is an example.
EDIT: for others coming across this answer in the future the right thing to do is:
don't write files in the first place, you never need to do this, and this question is typically asked by inexperienced folks who for some reason think that the only way to "save" a response before validation is to write it to a file. No, please don't waste your time - and please just match against the response. You can save it (or parts of it) to variables while you make other HTTP requests. And do not write your tests so that scenarios (or features) depend on other scenarios, this is a very bad practice. Also note that by default, Karate will dump all HTTP requests and responses in the log file (typically in target/karate.log) and also in the HTML report.
see if karate.write() works for you as per this answer
write a custom Java (or JS function that uses the JVM) to do what you want using Java interop
Also note that you can use karate.toCsv() to convert JSON into CSV if needed.
My justification for writing to a file is a different one. I am using karate explicitly to implement a mock. I want to expose an endpoint wherein the upstream system will send some basic data through json payload using POST/PUT method and karate will construct the subsequent payload file and stores it the specific folder, and this newly created payload file will be exposed through another GET call.
When I'm checking web requests in Chrome's DevTools on Mac (Network tab), I've got the Payload in the following format:
7|0|6|https://www.example.com/app/Basic/|00D1D071AC218DFE91521C012683E911|com.optionfair.client.common.services.nongenerated.RefreshService|getCometUpdates|I|J|1|2|3|4|3|5|6|6|173|VvAwAqy|o$UN|
which is basically separated by vertical bar character (|).
How I can copy or convert above payload from Chrome into some meaningful format such as JSON? Any ideas?
Btw. In this question it looks fine on the screenshot, but in my case, I don't have view parsed and it doesn't look like JSON format at all.
Using Google Chrome on Mac (Version 57.0.2987.133, 64-bit).
Reproducible steps:
Go to this page.
Open DevTools on Network/XHR tab and look for refresh requests.
My goal is to reuse/replicate the POST data in Request Payload in the command-line tool such as curl so it can be recognized (not necessary on the page mentioned above, but I'd like to know the general approach to deal with this blob format). I would expect JSON format, but it's not.
Here you need to look at the request header content-type to determine how this request was encoded before knowing what might parse it:
This is GWT RPC, so it can include serializations of built-in and custom Java Objects, where knowledge of the class is in both server-side Java and transpiled Java running on the client via Google Web Toolkit.
There is no reason for Chrome to understand this format directly, and it need not have a JSON or XML cannonicalization. Fully interpreting these calls to the extent it is possible on the client may require disassembly or introspection tricks against the transpiled client code, assuming the program wasn't transpiled with source maps.
Without digging into the client code, one can interpret the literal rpc while guessing at or probing the definition of classes and their methods by modifying the call.
I am working on a demo using MarkLogic to store emails exported from Outlook as XML, so that they stay searchable and accessible when I move away from Outlook.
I am using an AngularJS front-end calling either the native MarkLogic REST services of own REST services written in JAVA using Jersey.
MarkLogic SEARCH REST service works very well to get back a list of references to documents based on various search criteria, but I also want to display information stored inside the found documents.
I would like to avoid multiple REST calls and to get back only the needed information, so I am trying to use the EVAL REST service to run an xQuery.
It works well to get XML back (inside a multipart/mixed message) but I don't seem to be able to get JSON instead which would be much more convenient and is very easy with most other MarkLogic REST services.
I could use "json:transform-to-json()" in my xQuery or transform the XML to JSON in my JAVA code, but that does not look very elegant to me.
Is there a more efficient method to get where I am trying to go ?
First, json:transform-to-json seems plenty elegant to me. But of course it's not always the right answer.
I see three options you haven't mentioned.
server-side transforms - REST search supports server-side transforms which transform each document when you perform a bulk read by query. Those server-side transforms could generate any json you need.
search extract-document-data - this the simplest way to extract portions of documents. But it seems best if your documents are json to match your json response. Otherwise you get xml in your json response . . . unless you're ok with that.
custom search snippets - another very powerful way to customize what search returns
All of these options don't require the privileges that eval requires, which is a very good thing. Since eval allows execution of arbitrary code on your server, it requires special privileges and should be used with great care. Two other options before you use eval are (1) custom xquery installed in an http server, and (2) REST extensions.
The answers from Sam are what I would suggest. Specifically I would set a search option for search-extract-document-data (This is a search API option. If you are posting the request, then you can add the option in the XML you post back. If you are using GET, then you need to register the option ahead of time and call it. Relevant URLs to assist:
https://docs.marklogic.com/guide/rest-dev/search#id_48838
https://docs.marklogic.com/guide/search-dev/appendixa#id_44222
As for json.. ML8 will transform content. Use the accept-header or just add format=json to your results...
Example - xml which is what my content is stored as:
http://localhost:8000/v1/search?q=watermellon
...
<search:result index="1" uri="/sample/collections/1.xml" path="fn:doc("/sample/collections/1.xml")" score="34816" confidence="0.5982239" fitness="0.6966695" href="/v1/documents?uri=%2Fsample%2Fcollections%2F1.xml" mimetype="application/xml" format="xml">
<search:snippet>
<search:match path="fn:doc("/sample/collections/1.xml")/x">
<search:highlight>watermellon</search:highlight>
</search:match>
</search:snippet>
</search:result>
...
Example - json which is what my content is stored as:
http://localhost:8000/v1/search?q=watermellon&format=json
...
"index":1,
"uri":"/sample/collections/1.xml",
"path":"fn:doc(\"/sample/collections/1.xml\")",
"score":34816,
"confidence":0.5982239,
"fitness":0.6966695,
"href":"/v1/documents?uri=%2Fsample%2Fcollections%2F1.xml",
"mimetype":"application/xml",
"format":"xml",
"matches":[
{
"path":"fn:doc(\"/sample/collections/1.xml\")/x",
"match-text":[
{
"highlight":"watermellon"
}
]
}
]
}
...
For real heavy-lifting, you can use server-side transforms as in Sam's description. One note about this. Server-side transformations are not part of the search API, but part of the REST API. Just mentioning it so you have some idea of which tool you are using in each case..
I want to send text using POST method to another HTTP server and receive a JSON file in response from XQuery code of pipe in MarkLogic. The idea is whenever XML or JSON documents are inserted into MarkLogic, it triggers a pipe to read it and sends one element to another web server; in my situation, I want to send to Rosoka server to do natural language processing, after that I want to store the returned data "it is json file" in MarkLogic.
I appreciate if you could help.
marklogic only mentioned that is possible but no further help
There are many ways to do this.
A start is to look at "Triggers"
https://docs.marklogic.com/guide/app-dev/triggers
There is also Content Processing Framework ( "CPF" which is a higher level workflow based on triggers).
https://docs.marklogic.com/guide/cpf
if you read those guides you should be able to ask more specific questions if needed.
In our application we send requests to S&P like this:
http://SearchAndPromoteUrl/?parameter1=1¶meter2=2
My current understanding that S&P is external cloud service. sometimes we collect statistic and send to the server. Server analyzes our feed and we can get niformation from this S&P server.
1.Where can I found valid parameters after http://SearchAndPromoteUrl/?
2.Which manipulations should I execute for getting JSON instead of XML(S&P server should returns JSON Now our S&P returns xml).
This article says that it is possible
related information
http://microsite.omniture.com/t2/help/en_US/snp/8.15.0/SPguide.pdf
First I'd suggest using the updated documentation - https://marketing.adobe.com/resources/help/en_US/snp/8.16.0/c_getting_started.html
The recommended way to integrate with S&P is to use the search form they provide. This form takes care of the request and parameters for you.
UPDATE: Here are more details: https://marketing.adobe.com/resources/help/en_US/snp/t_copying_the_html_code_of_the_search_form_into_the_pages_of_your_website.html
Yes, you can use JSON instead of the standard XML response, and it's quite simple: just write your own presentation template using JSON instead of XML (see https://marketing.adobe.com/resources/help/en_US/snp/8.16.0/c_about_templates.html)
Have a nice day.