I am using the pycsw extension to produce a CSW file. I have harvested data from one CKAN instance [1], into another [2], and am now looking to run the pycsw 'paster load' command:
paster ckan-pycsw load -p /etc/ckan/default/pycsw.cfg -u [CKAN INSTANCE]
I get the error:
Could not pass xml doc from [ID], Error: Start tag expected, '<' not found, line 1, column 1
I think it is because when I visit this url:
[CKAN INSTANCE 2]/harvest/object/[ID]
It comes up with a JSON file as opposed to an XML (which it is expecting)
I have run the pycsw load command on other ckan instances and have had no problems with them. They also display an XML file at the url stated above, so I wanted to know how to get CKAN to serve an XML file instead of JSON?
Thanks in advance for any help!
As you've worked out, your datasets need to be in ISO(XML) format to load into a CSW server. A CKAN only has a copy of the dataset in ISO(XML) format if it harvested them from a CSW.
If you use the CKAN(-to-CKAN) harvester in the chain then the ISO(XML) record doesn't get transferred with it. So you'd either need to add this functionality to the CKAN(-to-CKAN) harvester, or get rid of the CKAN-to-CKAN harvest step.
Alternatively if the record originated in a CKAN, then it has no ISO(XML) version anyway, and you'd need to create that somehow.
Related
I have a basic NIFI flow with one GenerateFlowFile processor and one LookupRecord processor with RestLookupService.
I need to do a rest lookup and enrich the incoming flow file with rest response.
I am getting errors that the lookup service is unable to lookup coordinates with the value I am extracting from the incoming flow file.
GenerateFlowFile is configured with simple JSON
LookupRecord is configured to extract the key from the JSON and populate it to the RestLookupService. Also, JsonReader and JsonSetWriter is configured to read the incoming flow file and to write the response back to the flow file
The RestLookupService itself exits with JsonParseException about unexpected character '<'
RestLookupService is configured with my API running in the cloud in which I am trying to use the extracted KEY from the incoming flow file.
The most interesting bug is that when I configure the URL to point for example to mocky.io everything works correctly so it means that the issue itself is tight with the API URL I am using (http://iotosk.ddns.net:3006/devices/${key}/getParsingData). I have tried also removing the $key, using the exact URL, using different URLs..
Of course the API is working OK over postman/curl anything else. I have checked the logs on the container that the API is running on and there is no requests in the logs what means that nifi is failing even before reaching the API. At least on application level...
I am absolutely out of options without any clue how to solve this. And with nifi also google is not helpful.
Does anybody see any issue in the configuration or can point me in some direction what can cause this issue?
After some additional digging. The issue was connected with authentication logic even
before the API receives it and that criples the request and and returned XML as Bryan Bende suggested in the comment.
But definitely better logging in nifi will help to solve this way faster...
My Thread Group contains "Include Controller" with a reference to jmx and a "Simple Controller" with http requests.
From Command line(Non GUI mode), While I execute,
I need CSV results to contain results only from "Simple Controller"
OR
a way to identify response time based on Controller name in the generated CSV
Can you please help - I am passing below code in the command line
sh Jmeter.sh -n -t R3Functions.jmx -l Results/Calendar.csv -J ucount=10
From the generated csv, I am finding it very difficult to get the
response time of http requests only under "Simple Controller"
as it also includes results of http requests from "Include Controller".
I'm not aware of any existing approaches to filter results basing belonging to this or that parent controller so I would recommend just use different labels for the requests originating from main and included .jmx files so you will be able to distinguish them either manually or using Filter Results Tool plugin which can filter sample results basing on regular expression matches of samplers labels (names).
You can install Filter Results Tool using JMeter Plugins Manager
I have created jmeter script for load testing and added CSV data element to read Username and Password of 300 Users.Also followed below steps;
Created CSV file using a notepad and stored it in a directory where created script stored.
Defined variables and file path in CSV data set element
Used variables as parameters of HTTP request
Increased number of Threads
But it doesn't worked as expected.How can i fix this?
Temporary reduce threads count to 1-2, add View Results Tree listener and inspect request and response details.
If don't see proper username/password combination in the request body - your script is misconfigured. Check out Using CSV DATA SET CONFIG guide for detailed setup instructions.
If you see valid username/password pair but login is not successful, follow below checklist:
Add HTTP Cookie Manager to your test plan
If point 1 doesn't help - check for dynamic request parameters. Probably you'll need to extract some value from the first response (when you execute GET request for the login page) with the Regular Expression Extractor, convert it into a JMeter Variable and add it as a parameter to the request which performs login.
I am trying to use SpringXD to stream some JSON metrics data to a Oracle database.
I am using this example from here: SpringXD Example
Http call being made: EarthquakeJsonExample
My shell cmd.
stream create earthData --definition "trigger|usgs| jdbc --columns='mag,place,time,updated,tz,url,felt,cdi,mni,alert,tsunami,status,sig,net,code,ids,souces,types,nst,dmin,rms,gap,magnitude_type' --driverClassName=driver --username=username --password --url=url --tableName=Test_Table" --deploy
I would like to capture just the properties portion of this JSON response into the given table columns. I got it to the point where it doesn't give me a error on the hashing but instead just deposits a bunch of nulls into the column.
I think my problem is the parsing of the JSON itself. Since really the properties is in the Features array. Can SpringXD distinguish this for me out of the box or will I need to write a custom processor?
Here is a look at what the database looks like after a successful cmd.
Any advice? Im new to parsing JSON in this fashion and im not really sure how to find more documentation or examples with SpringXD itself.
Here is reference to the documentation: SpringXD Doc
The transformer in the JDBC sink expects a simple document that can converted to a map of keys/values. You would need to add a transformer upstream, perhaps in your usgs processor or even a separate processor. You could use a #jsonPath expression to extract the properties key and make it the payload.
I got an input xml file like this :
<root>
<item ...>
<item ...>
<item ...>
</root>
and I need to construct and send messages in 2 ways in the same orchestration :
1. Send to one destination as 1 xml file per item
2. Send the whole bunch of "item" as one single csv flat file.
My file is actually handled by a pipeline that extracts every "item" from the envelope. The problem is that I need to merge all "items" based on a certain condition.
Any ideas how to achieve this ?
There seem to be at least 2 ways of going about this - it isn't clear as to how you 'arrive' at the input xml file batch - this will drive the decision, IMO.
Since it seems that you've already got all the messages in a single xml batch at the start, this should be quite easy. Before you debatch them in a pipeline, you need to ensure that you also publish this batch message (root ...) into the messagebox if it isn't already so (i.e. Direct binding, if the message doesn't already come from the messagebox).
You can then create a map for the CSV file which takes the root message as input and then filters out the items which you don't want in the CSV. To do the filtering in the Map, you could use a looping functoid with conditionals, or my preference would just to be to implement the map in xslt and then apply templates just to desirable items with an xpath filter. A subscribing FILE send port, which filters (BTS.MessageType) to the incoming xml batch message can then apply this map.
The individual xml files would then be processed by your debatching pipeline and then another subscribing Physical FILE send port can write them out.
2 . Alternatively, if it is too late and the root xml file has been debatched (and you can't get to the original Xml file FWR), you would need to use another orch to reassemble the messages needed for the CSV (scatter and gather). This will be more complicated, as you will likely need to correlate the messages (e.g. on some batch identifier), or apply a timer, etc.
Have a look at the Pipeline Aggregator sample on how to collect the 'desirable' CSV messages into a Microsoft.XLANGs.Pipeline.SendPipelineInputMessages variable using a loop, and then use a pipeline to assemble the batch. If the criteria for 'desirable' are already promoted on the individual item messages, then you can apply the filter on your receive, but if not, you will need to use a decision in your loop to determine whether or not to add the message to the batch.