I am using ZappySys for SSIS ZS JSON Source (Rest API or File) to import data from on online service. I have been successful to Load the data into OLE DB destination but so far only 1000 rows are imported out of 3000 rows.
I tried pagination option to import the next offset data but so far, I have been unsuccessful in it.
Offset or page no. is not mentioned in the URL so that method won't work. JSON format is something like this:
<response>
<result>
<data>
<data>
.
.
<data>
<total_records>3000</total_records>
<start_offset>0</start_offset>
<next_offset>1000</next_offset>
</result>
</response>
I yet have no idea to how to import all 3000 rows. My guess is to use POST option either in settings or Pagination to get the next 1000 rows and so on. I tried POST with pagination but I couldn't make it work.
I work for ZappySys
Have you looked at our article about REST API Pagination (Method #3 and #4)?
You can extract next_offset and then Pass it to the URL of the next Call. Make sure to configure pagesize and incrementby parameters correctly.
You can open a support ticket with ZappySys if you need help (Go to zappysys.com/support and submit an inquiry there and we will help you to configure).
Related
I am using SSIS 2010 and using a test extension ZappySys connecting test JSON Source (rest API or File).
The issue I have is the total rows to scan 300 default. I have tried to override this, and it still returned 300 rows. I would like to use this extension more, but is there a way of getting more than 300 rows of data? Does anyone know how to by pass this?
There is this post on zappysys blog but does not state how to get more than 300 rows.
https://zappysys.com/blog/how-to-read-data-from-servicenow-rest-api-ssis/
Disclaimer: I work for ZappySys.
Scan option affect only Metadata Guess in ZappySys. Based on your description Looks like your issue is not metadata guess but fetching more rows with pagination. Process of looping through ServiceNow data is described here
For using pagination in ZappySys make sure you configure JSON source as per the article below
Step-By-Step : Configure ServiceNow API Pagination
Screenshot - Pagination Settings:
Here is an example of Pagination in ServiceNow. Let's say you fetching all rows for MyTable1. In such case you have to keep calling API like below until last page is reached. You detect last page by trapping WebException with StatusCode 404
/api/now/table/MyTable1?sysparm_limit=10&sysparm_offset=0
/api/now/table/MyTable1?sysparm_limit=10&sysparm_offset=10
/api/now/table/MyTable1?sysparm_limit=10&sysparm_offset=20
/api/now/table/MyTable1?sysparm_limit=10&sysparm_offset=30
/api/now/table/MyTable1?sysparm_limit=10&sysparm_offset=NNN ...Last page reached you get 404 error. Stop looping
ZappySys Pagination settings takes care of these for you automatically.
I am using the pycsw extension to produce a CSW file. I have harvested data from one CKAN instance [1], into another [2], and am now looking to run the pycsw 'paster load' command:
paster ckan-pycsw load -p /etc/ckan/default/pycsw.cfg -u [CKAN INSTANCE]
I get the error:
Could not pass xml doc from [ID], Error: Start tag expected, '<' not found, line 1, column 1
I think it is because when I visit this url:
[CKAN INSTANCE 2]/harvest/object/[ID]
It comes up with a JSON file as opposed to an XML (which it is expecting)
I have run the pycsw load command on other ckan instances and have had no problems with them. They also display an XML file at the url stated above, so I wanted to know how to get CKAN to serve an XML file instead of JSON?
Thanks in advance for any help!
As you've worked out, your datasets need to be in ISO(XML) format to load into a CSW server. A CKAN only has a copy of the dataset in ISO(XML) format if it harvested them from a CSW.
If you use the CKAN(-to-CKAN) harvester in the chain then the ISO(XML) record doesn't get transferred with it. So you'd either need to add this functionality to the CKAN(-to-CKAN) harvester, or get rid of the CKAN-to-CKAN harvest step.
Alternatively if the record originated in a CKAN, then it has no ISO(XML) version anyway, and you'd need to create that somehow.
I'm brand new to Pentaho and I'm trying to do the following workflow:
read a bunch of lines out of a DB
do some transformations
POST them to a REST web service in JSON
I've got the first two figured out using an input step and the Json Output step.
However I have two problems doing the final step:
1) I can't get the JSON formatted how I want. It insists on doing {""=[{...}]} when I just want {...}. This isn't a big deal - I can work around this since I have control over the web service and I could relax the input requirements a bit. (Note: this page http://wiki.pentaho.com/display/EAI/JSON+output gives an example for the output I want by setting no. rows in a block=1 and an empty JSON block name, but it doesn't work as advertised.)
2) This is the critical one. I can't get the data to POST as JSON. It posts as key=value, where the key is the name I specify in the HTTP Post field name (on the 'Fields' tab) and the value is the encoded JSON. I just want to post the JSON as the request body. I've tried googling on this but can't find anyone else doing it, leading me to believe that I'm just approaching this wrong. Any pointers in the right direction?
Edit: I'm comfortable scripting (in Javascript or another language) but when I tried to use XmlHttpRequest in a custom javascript snippet I got an error that XmlHttpRequest is not defined.
Thanks!
This was trivial...just needed to use the REST Client (http://wiki.pentaho.com/display/EAI/Rest+Client) instead of the HTTP Post task. Somehow all my googling didn't discover that, so I'll leave this answer here in case someone else has the same problem as me.
You need to parse the JSON using a Modified JavaScript step. e.g. if the Output Value from the JSON Output is called result and its contents are {"data"=[{...}]}, you should call var plainJSON = JSON.stringify(JSON.parse(result).data[0]) to get the JSON.
In the HTTP Post step, the Request entity field should be plainJSON. Also, don't forget to add a header for Content-Type as application/json (you might have to add that as a constant)
Here's my situation:
I want to do this:
I have a list of URLs in a MySQL database which I want to hit using a HTTP Request to see if the response is a HTTP Status code of 404 or not.
I have done this:
Added and configured a JDBC Config Element.
Added and configured a JDBC Request Sampler. Basically a select statement that returns a table with 8 columns. I have provided 8 comma-separated variables for the 'Variable names' field, so that the results of the JDBC request can be identified with these variable names.
Created a HTTP Request Sampler that uses one of those variables ${url} in the 'Server Name or IP' field.
Though the JDBC request works flawlessly and returns a table with a bunch of rows, the problem with this is that the HTTP Request Sampler never picks up the variable from the JDBC Request result.
The HTTP Request looks like this in the 'View Results Tree':
GET http://${url}/
I have tried these solutions:
Add 'Save Responses to a File' listener to the JDBC Request. This creates a file of type '.plain' and not a CSV. Had it been a CSV, I could have utilized that CSV file by creating a CSV Data Set Config. So this attempt failed.
I have tried forcing the file name in the above attempt to always use 'C:\JMETERTest\data.csv'. But it ends up creating a new file named 'C:\JMETERTest\data.csv1.plain'. This attempt failed too.
I tried to reference the URL column as ${url_1} in the HTTP Request's Server Name field. It worked. But the problem now is that in the results tree, all the requests are going for the the URL from only the first row of the result set. I see that this is because of the row number '_1' specified in the ${url_1} above. I can use this if someone can suggest a way to parameterize the '_1' into a variable that I can loop through (probably using a 'Counter' element). I created a Counter Config Element by the Reference Name 'loopCounter'. And used this in the Server Name field of the HTTP Request:
${url_("${loopCounter}")}
But now my HTTP Requests look lamer:
GET http://${url_("${loopCounter}")}/
This did not work too.
Solution 3 looks more doable to be only if I could resolve the parameterization of the row number.
I am open to JMeter Plugin suggestions too.
I will update anything else that I try as we go on.
P.S. Please let me know if my question is not clear in anyway.
Have you tried wrapping the HTTP sampler in a ForEach controller (parent) where the variable for the controller is the URL variable obtained from the JDBC sampler?
Also, the output variable in the ForEach will be the variable you now use in the HTTP sampler.
That way it will iterate through each variable from the beginning of the index to the end and run the sampler once each time.
In 'Save responses to a File' Listener, Select Checkboxes "Don't add Suffix and Prefix". Checking these two options will ensure, you get exact Log file name.
Hy!
I need to know how can I filter/parse some json data into a JSP.
It would be in order to create a kind of 'little' API.
I have made one test via PHP, the thing goes like that.
I send a parameter to an URL like this
records.php?artist='tame impala'
In my php file first I proceed to trim the parameter artist, then I compare to a JSON will all the records and finally I print only the records from 'tame impala'.
Is it possible to do that in a JSP file without any servlet or similar?
Thanks a lot!