I've a JSON response based on a webservice request.
[
{
"type": " --T::00"
},
{
"address": "10049 College Way N",
"longitude": "-122.335022",
"latitude": "47.701756",
"incident_number": "F110104009",
"type": "Aid Response",
"report_location": {
"needs_recoding": false,
"longitude": "-122.335022",
"latitude": "47.701756"
}
},
{
"address": "5929 Beach Dr Sw",
"longitude": "-122.397816",
"latitude": "47.550431",
"incident_number": "F110104008",
"type": "Aid Response",
"report_location": {
"needs_recoding": false,
"longitude": "-122.397816",
"latitude": "47.550431"
}
}
Is there a way to manipulate the payload to remove this from the header {
"type": " --T::00"
} or just add it in the footer. I currently use a JSON to Object transformer and added java.util.List. I've seen posts of using a groovy transformer to add to the message but how about removing elements such as the header?
Thanks!
What you can do is :-
1. Extract all the elements value from the JSON request using <json:json-to-object-transformer/> and store them in some variables.
2. Try to build your own JSON request using Mule Expression Transformer and assign the values from variable. Thus, you can create a Dynamic JSON you require..
You can check here how to build Dynamic JSON using Expression Transformer :-
How to transform json-to-json document in Mule ESB
Related
I have to extract attributes from a json file that I receive from an api call using InvokeHTTPCustom. JSON FILE has the following sample data :
[
{
"input_index": 0,
"candidate_index": 0,
"delivery_line_1": "1 Santa Claus Ln",
"last_line": "North Pole AK 99705-9901",
"delivery_point_barcode": "997059901010",
"components": {
"primary_number": "1",
"street_name": "Santa Claus",
"street_suffix": "Ln",
"city_name": "North Pole",
"state_abbreviation": "AK",
"zipcode": "99705",
"plus4_code": "9901",
"delivery_point": "01",
"delivery_point_check_digit": "0"
},
"metadata": {
"record_type": "S",
"zip_type": "Standard",
"county_fips": "02090",
"county_name": "Fairbanks North Star",
"carrier_route": "C004",
"congressional_district": "AL",
"rdi": "Commercial",
"elot_sequence": "0001",
"elot_sort": "A",
"latitude": 64.75233,
"longitude": -147.35297,
"coordinate_license": 1,
"precision": "Rooftop",
"time_zone": "Alaska",
"utc_offset": -9,
"dst": true
},
"analysis": {
"dpv_match_code": "Y",
"dpv_footnotes": "AABB",
"dpv_cmra": "N",
"dpv_vacant": "N",
"dpv_no_stat": "Y",
"active": "Y",
"footnotes": "L#"
}
},
{
"input_index": 1,
"candidate_index": 0,
"addressee": "Apple Inc",
"delivery_line_1": "1 Infinite Loop",
// truncated for brevity
}
]
I have extracted all the required data such as address, state, city, primary_number, etc.
However, when I try to extract latitude,longitude from metadata, it leads to failure in EvaluateJsonPathAttributeCustom processor. Other attributes, which are in string format, get extracted correctly. However, this being not a string, might be issue, is my diagnosis.
How do I get this working?
I need to extract longitudes and latitudes.
Please give detail explanation as I am new to nifi.
Configuration in nifi for EvaluateJsonPathAttributeCustom:
Attribute Name Input : x**.json
Attribute Name Output : latitude
JsonPathExpresssion : $[0].metadata.latitude
Splitif.. : False
One way to do this is by using the JOLT https://jolt-demo.appspot.com/.
I would recommend using the JoltTransformJSON NiFi Processor as it can really help make things easy to pull out only the data that you want. I have tried your specific request and it will work to pull out those data. You can configure JOLT to pull any data you require and it might be easier once you get the hang of it.
[{
"operation": "shift",
"spec": {
"*": {
"metadata": {
"latitude": "latitude",
"longitude": "longitude"
}
}
}
}]
I am doing an API call every 40 mins to retrieve the current status information of every car in a car fleet. And each call adds one new JSON document to a Cloudant database. Each JSON document defines the current availability status for every car across many locations in many cities. There are currently around 2200 JSON documents in the database. All JSON documents have one field called payload that contains all information; it is a large array of objects. Instead of retrieving the whole payload array of objects I would like to retrieve only the needed info with a query (so, only one or several objects of that array). However, I have difficulty drafting a query that results only in the needed data.
Below, I'll explain my problem in more detail:
When saving the JSON document to Cloudant, a timestamp is defined in the document. The _id parameter is defined to be equal to this timestamp. Below, I show a simplified version of these JSON documents:
{
"_id": "1540914946026",
"_rev": "3-c1834c8a230cf772e41bbcb9cf6b682e",
"timestamp": 1540914946026,
"datetime": "2018-10-30 15:55:46",
"payload": [
{
"cityName": "Abcoude",
"locations": [
{
"address": "asterlaan 28",
"geoPoint": {
"latitude": 52.27312,
"longitude": 4.96768
},
"cars": [
{
"mod": "BMW",
"state": "FREE"
}
]
}
],
"availableCars": 1,
"occupiedCars": 0
},
{
"cityName": "Alkmaar",
"locations": [
{
"address": "Aert de Gelderlaan 14",
"geoPoint": {
"latitude": 52.63131,
"longitude": 4.72329
},
"cars": [
{
"model": "Volswagen",
"state": "FREE"
}
]
},
{
"address": "Ardennenstraat 49",
"geoPoint": {
"latitude": 52.66721,
"longitude": 4.76046
},
"cars": [
{
"mod": "BMW",
"state": "FREE"
}
]
},
{
"address": "Beneluxplein 7",
"geoPoint": {
"latitude": 52.65356,
"longitude": 4.75817
},
"cars": [
{
"mod": "BMW",
"state": "FREE"
}
]
},
{
"address": "Dr. Schaepmankade 1",
"geoPoint": {
"latitude": 52.62595,
"longitude": 4.75122
},
"cars": [
{
"mod": "BMW",
"state": "OCCUPIED"
}
]
},
{
"address": "Kennemerstraatweg",
"geoPoint": {
"latitude": 52.62909,
"longitude": 4.74226
},
"cars": [
{
"model": "Mercedes",
"state": "FREE"
}
]
},
{
"address": "NS Station Alkmaar Noord/Parkeerterrein Noord",
"geoPoint": {
"latitude": 52.64366,
"longitude": 4.7627
},
"cars": [
{
"model": "Tesla",
"state": "FREE"
}
]
},
{
"address": "NS Station Alkmaar/Stationsweg 56",
"geoPoint": {
"latitude": 52.6371,
"longitude": 4.73935
},
"cars": [
{
"model": "Tesla",
"state": "FREE"
}
]
},
{
"address": "Oude Hoeverweg",
"geoPoint": {
"latitude": 52.63943,
"longitude": 4.72928
},
"cars": [
{
"model": "Tesla",
"state": "FREE"
}
]
},
{
"address": "Parkeerterrein Wortelsteeg",
"geoPoint": {
"latitude": 52.63048,
"longitude": 4.75487
},
"cars": [
{
"model": "Tesla",
"state": "OCCUPIED"
}
]
},
{
"address": "Schoklandstraat 38",
"geoPoint": {
"latitude": 52.65812,
"longitude": 4.75359
},
"cars": [
{
"model": "Volkswagen",
"state": "FREE"
}
]
}
],
"availableCars": 8,
"occupiedCars": 2
}
]
}
As you can see, the payload field is an array that has several objects (FYI: every object in this array represents one specific city: there are 1600 cities, so 1600 nested objects inside the payload array). Furthermore, inside each of the 1600 objects mentioned, other arrays and objects are again nested inside. For all objects in the payload array, the first field is cityName.
Furthermore, there is a nested array locations (inside each of the 1600 objects of the payload array) representing all addresses in a specific city. The locations array can be of size 1 to 600, meaning 1 to 600 nested objects / addresses per city. The last two fields in all objects of the payload array are availableCars and occupiedCars.
I want query documents to see how many cars are available and occupied for a specific city during a specific time interval. To do this:
I have to specify a start timestamp (or id) and an end timestamp, resulting in only the JSON documents within this interval.
Furthermore, I will need to specify inside the JSON documents only one or more specific cities by cityName (there are 1600 cities) and then get the number of available cars availableCars and the number of occupiedCars for those cities.
For example, in this simplified example, I would like to query for the status information (availableCars & `occupiedCars) for the city of Alkmaar from 1540914946026 (epoch time) until now. I would like to get the following result:
{
"id":"1540914946026",
"cityName":"Alkmaar",
"availableCars":8,
"occupiedCars":2
}
This is just an example, in reality, I want to be able to query for other cities as well, or query for several cities together and then get for each of those cities the number of available cars availableCars and the number of occupied cars occupiedCars.
Could anyone help me to define a query and index to be able to get the above result? Can I do this with cloudant query?
Your data model does not play to Cloudant's strengths. Let each document group data that changes and is accessed together. Your items in your payload array would be much better stored as discrete documents.
If you find yourself reaching into growing arrays inside documents for subsets of data, this is a warning sign that your data model is not ideal: the document is now mutable and growing (with potential update conflicts as a result), and access becomes more cumbersome over time as Cloudant has no mechanism to only retrieve parts of a document. Moreover, Cloudant has a limit (1M) on document size, so by using your proposed model, you will likely hit that limit, too, and your application would stop working.
With that said, it is possible to create a view index that lets you emit each component of your payload, which would let you look up data per city -- but that solution is still subject to all the limitations above (document model is mutable, documents grow large etc).
Rule of thumb: small documents. Immutable model, where possible. Documents group data that either change, or are accessed as a unit.
I have the following file which I want to use as test data for a project. The only data I need from the file is the latitude longitude data in Location objects.
I managed to delete all unwanted data from the file using the following command:
sed -i '' '/pattern/d' ./locations.json
Now I am stuck trying to remove the unwanted , character on each longitude line. How can I quickly do this using shell script?
[
{
"latitude":-35.2777,
"longitude":149.1189,
},
{
"latitude":-35.3049,
"longitude":149.1412,
},
{
"latitude":-35.1627,
"longitude":150.6907,
},
\\ and so on
]
What I am trying to do is replace this patters "longitude":\d+.\d+, with "longitude":\d+.\d+
jq is the right tool for processing JSON data:
jq 'map({latitude, longitude})' ./locations.json
The output (shortened):
[
{
"latitude": -35.2777,
"longitude": 149.1189
},
{
"latitude": -35.3049,
"longitude": 149.1412
},
{
"latitude": -35.1627,
"longitude": 150.6907
},
{
"latitude": -35.028,
"longitude": 150.5501
},
{
"latitude": -35.1333,
"longitude": 150.7
},
....
]
https://stedolan.github.io/jq/manual/v1.5/
I am new to Liftweb. I want to Store a Json File in Mysql database using Lift Mapper
My Json File Like Below:-
[
{
"name": "Root Category",
"Id": "1",
"dispName": "",
"childs": [
{
"name": "Sub Category",
"Id": "",
"dispName": "",
"childs": [
{
"name": "Spec1",
"Id": "",
"dispName": "",
"childs": []
}
]
}
]
},
{
"name": "Root Category",
"Id": "",
"dispName": "",
"childs": [
{
"name": "Sub Category",
"Id": "",
"dispName": "",
"childs": [
{
"name": "Spec1",
"Id": "",
"dispName": "",
"childs": []
}
]
}
]
}
]
Is it Possible to store a Json File in Lift Mapper .Please give me Suggestions. It will be great if some one provide any sample
Best Regards
GSY
At the moment there is no good support for storing JSON in MySQL. I mean it's not going to provide capabilities MongoDB provides for example. However there are some JSON processing functions provided by community if you want. Given all that you can store it in VARCHAR. TEXT or BLOB field type as simple text. Here is a Mapper example:
import net.liftweb.mapper._
import net.liftweb.common._
class SomeDbClass extends LongKeyedMapper[SomeDbClass] with IdPK {
def getSingleton = SomeDbClass
// set limit of chars - can be used in `validate()`
object quota_type extends MappedString(this, 1024)
}
object SomeDbClass extends SomeDbClass with LongKeyedMetaMapper[SomeDbClass]
For one of my projects I store JSON as a string in Postgres similarly because I just need to read and write it without having to parse it in DB and query by fields. Whenever I need efficient JSON storage with query and update support I use MongoDB with Record + ( Casbah or Rogue ).
I am quite new to json and webservices.
I have a question which might as well be a dumb one but I have not been able to find an answer.
Lets say I have a html form which performs a GET method on submission.
The server performs an HTTP response.
How does one make sure that the content(body) of the response is in JSON format?
-V
This is quite a vague question as I don't know what your server is doing and replying (obviously it should be in the JSON format response) but you should also include a Content-type HTTP header as shown below so the browser knows it is receiving JSON:
Content-type: application/json
Your JSON response should look like this (taken from Wikipedia):
{
"firstName": "John",
"lastName": "Smith",
"age": 25,
"address": {
"streetAddress": "21 2nd Street",
"city": "New York",
"state": "NY",
"postalCode": 10021
},
"phoneNumbers": [
{
"type": "home",
"number": "212 555-1234"
},
{
"type": "fax",
"number": "646 555-4567"
}
]
}
data=response_from_the_page;
try {
json = $.parseJSON(data);
} catch (e) {
// not json
}