NYC Open data DOB Missing Information - socrata

I am facing some issue in NYC department of building API.
help me if you know any other API giving the same information
I have used this API but didn't work for me
https://data.cityofnewyork.us/resource/83x8-shf7.json
Missing fields
permitee detailed address
https://data.cityofnewyork.us/resource/83x8-shf7.json?$where=filing_date BETWEEN '2018-05-01T06:00:00' AND '2018-05-30T10:00:00'
Also i am not able get expected data using filters for "filing_date" from same api
expected data should return all data between 2018-05-01 and 2018-05-30 for this API But i am getting only few results.
I am getting this data
[
{
"bin__": "3118313",
"bldg_type": "1",
"block": "05143",
"borough": "BROOKLYN",
"city": "BROOKLYN",
"community_board": "314",
"dobrundate": "2018-05-03T00:00:00.000",
"expiration_date": "2018-06-11T00:00:00.000",
"filing_date": "2018-05-02T00:00:00.000",
"filing_status": "INITIAL",
"gis_census_tract": "1522",
"gis_council_district": "40",
"gis_latitude": "40.641731",
"gis_longitude": "-73.966432",
"gis_nta_name": "Flatbush",
"house__": "328",
"issuance_date": "2018-05-02T00:00:00.000",
"job__": "321679046",
"job_doc___": "01",
"job_start_date": "2018-05-02T00:00:00.000",
"job_type": "A2",
"lot": "00068",
"non_profit": "N",
"owner_s_business_name": "N/A",
"owner_s_business_type": "INDIVIDUAL",
"owner_s_first_name": "MATTHEW",
"owner_s_house__": "328",
"owner_s_house_street_name": "ARGYLE ROAD",
"owner_s_last_name": "LIMA",
"owner_s_phone__": "3475968096",
"owner_s_zip_code": "11218",
"permit_sequence__": "01",
"permit_si_no": "3452932",
"permit_status": "ISSUED",
"permit_subtype": "OT",
"permit_type": "EW",
"permittee_s_business_name": "BMB BUILDER INC",
"permittee_s_first_name": "YUAN HANG",
"permittee_s_last_name": "XIAO",
"permittee_s_license__": "0612790",
"permittee_s_license_type": "GC",
"permittee_s_phone__": "9175776544",
"residential": "YES",
"self_cert": "N",
"site_fill": "NOT APPLICABLE",
"state": "NY",
"street_name": "ARGYLE ROAD",
"superintendent_business_name": "BMB BUILDER INC",
"superintendent_first___last_name": "YUAN HANG XIAO",
"work_type": "OT",
"zip_code": "11218"
}]
Expected Data should be
[{
"bin__": "1090379",
"bldg_type": "2",
"block": "00760",
"borough": "MANHATTAN",
"city": "GREAT NECK",
"community_board": "104",
"dobrundate": "2018-05-02T00:00:00.000",
"expiration_date": "2018-10-28T00:00:00.000",
"filing_date": "2018-05-01T00:00:00.000",
"filing_status": "RENEWAL",
"gis_census_tract": "111",
"gis_council_district": "3",
"gis_latitude": "40.753978",
"gis_longitude": "-73.993673",
"gis_nta_name": "Hudson Yards-Chelsea-Flatiron-Union Square",
"house__": "337",
"issuance_date": "2018-05-01T00:00:00.000",
"job__": "121187606",
"job_doc___": "01",
"job_start_date": "2016-02-19T00:00:00.000",
"job_type": "NB",
"lot": "00020",
"non_profit": "N",
"owner_s_business_name": "HKONY WEST 36 LLC",
"owner_s_business_type": "PARTNERSHIP",
"owner_s_first_name": "SAM",
"owner_s_house__": "420",
"owner_s_house_street_name": "GREAT NECK ROAD",
"owner_s_last_name": "CHANG",
"owner_s_phone__": "9178380886",
"owner_s_zip_code": "11021",
"permit_sequence__": "07",
"permit_si_no": "3451790",
"permit_status": "ISSUED",
"permit_type": "NB",
"permittee_s_business_name": "OMNIBUILD CONSTRUCTION IN",
"permittee_s_first_name": "PETER",
"permittee_s_last_name": "SERPICO",
"permittee_s_license__": "0608390",
"permittee_s_license_type": "GC",
"permittee_s_phone__": "2124191930",
"self_cert": "N",
"site_fill": "ON-SITE",
"site_safety_mgr_s_first_name": "ROBERT",
"site_safety_mgr_s_last_name": "FILIPPONE",
"special_district_1": "GC",
"state": "NY",
"street_name": "W 36 ST",
"zip_code": "10018"
}]

Combing through the JSON, it appears that these columns are not matching: permit_subtype, superintendent_business_name, superintendent_first___last_name, site_safety_mgr_s_first_name, site_safety_mgr_s_last_name, and special_district_1.
Looking at the original data sources, the columns that do not match are instances where the field is blank for that field. That is, bin__ = 1090379 does not have a permit_subtype, so it is omitted in the JSON (which is standard practice).
It will, however, be included in the CSV output since that format must include all columns: https://data.cityofnewyork.us/resource/83x8-shf7.csv?$where=filing_date%20BETWEEN%20%272018-05-01T06:00:00%27%20AND%20%272018-05-30T10:00:00%27.
This answer took a bit of digging because it wasn't immediately obvious which columns were different between the two examples. It's always helpful to over-explain to make it easier to track-down the issue.
Likewise, per the filing_date question, please include the query you're attempting to use.

Related

EvaluateJsonPathAttributeCustom - Nifi

I have to extract attributes from a json file that I receive from an api call using InvokeHTTPCustom. JSON FILE has the following sample data :
[
{
"input_index": 0,
"candidate_index": 0,
"delivery_line_1": "1 Santa Claus Ln",
"last_line": "North Pole AK 99705-9901",
"delivery_point_barcode": "997059901010",
"components": {
"primary_number": "1",
"street_name": "Santa Claus",
"street_suffix": "Ln",
"city_name": "North Pole",
"state_abbreviation": "AK",
"zipcode": "99705",
"plus4_code": "9901",
"delivery_point": "01",
"delivery_point_check_digit": "0"
},
"metadata": {
"record_type": "S",
"zip_type": "Standard",
"county_fips": "02090",
"county_name": "Fairbanks North Star",
"carrier_route": "C004",
"congressional_district": "AL",
"rdi": "Commercial",
"elot_sequence": "0001",
"elot_sort": "A",
"latitude": 64.75233,
"longitude": -147.35297,
"coordinate_license": 1,
"precision": "Rooftop",
"time_zone": "Alaska",
"utc_offset": -9,
"dst": true
},
"analysis": {
"dpv_match_code": "Y",
"dpv_footnotes": "AABB",
"dpv_cmra": "N",
"dpv_vacant": "N",
"dpv_no_stat": "Y",
"active": "Y",
"footnotes": "L#"
}
},
{
"input_index": 1,
"candidate_index": 0,
"addressee": "Apple Inc",
"delivery_line_1": "1 Infinite Loop",
// truncated for brevity
}
]
I have extracted all the required data such as address, state, city, primary_number, etc.
However, when I try to extract latitude,longitude from metadata, it leads to failure in EvaluateJsonPathAttributeCustom processor. Other attributes, which are in string format, get extracted correctly. However, this being not a string, might be issue, is my diagnosis.
How do I get this working?
I need to extract longitudes and latitudes.
Please give detail explanation as I am new to nifi.
Configuration in nifi for EvaluateJsonPathAttributeCustom:
Attribute Name Input : x**.json
Attribute Name Output : latitude
JsonPathExpresssion : $[0].metadata.latitude
Splitif.. : False
One way to do this is by using the JOLT https://jolt-demo.appspot.com/.
I would recommend using the JoltTransformJSON NiFi Processor as it can really help make things easy to pull out only the data that you want. I have tried your specific request and it will work to pull out those data. You can configure JOLT to pull any data you require and it might be easier once you get the hang of it.
[{
"operation": "shift",
"spec": {
"*": {
"metadata": {
"latitude": "latitude",
"longitude": "longitude"
}
}
}
}]

Import json array of objects to Elastic Search App

I have following sample json data with structure
[{
"URL": "http://www.just-eat.co.uk/restaurants-bluebreeze-le3/menu",
"_id": "55f14313c7447c3da7052518",
"address": "56 Bonney Road",
"address line 2": "Leicester",
"name": "Blue Breeze Fish Bar",
"outcode": "LE3",
"postcode": "9NG",
"rating": 5.5,
"type_of_food": "Fish \u0026 Chips"
},
{
"URL": "http://www.just-eat.co.uk/restaurants-bluebreeze-le3/menu",
"_id": "55f14313c7447c3da7052519",
"address": "56 Bonney Road",
"address line 2": "Leicester",
"name": "Blue Breeze Fish Bar",
"outcode": "LE3",
"postcode": "9NG",
"rating": 5.5,
"type_of_food": "Fish \u0026 Chips"
},
...... and so on thousands of objects
]
So i am trying to import this json data to Elastic Search App cluster i created... But i get error in all documents as follows
Indexing Summary
Something went wrong. Please address the errors and try again.
2548 documents with errors...
they didn't give any error details... just said listed all my objects with "semicolon highlighted"...
Can anyone tell me why am i not able to upload the data?

How would I query nested JSON data values in SQL using json_extract()?

Mysql version is 6.7+
Example of the column I need to get a value from below.
json_extract(goal_templates.template_data, group_concat('$.resources'))
->This results in a NULL return for all rows.
:template_data
{"Resolve housing issues": {"tasks": [{"name": "Use Decision Map to explore paths to resolving'' housing issue", "end_date": "15 days", "taskType": 3, "help_text": "", "resources": [], "start_date": "today", "actionOrder": "1", "recur_every": "", "resource_reference_id": ""}, {"name": "Select a local Debt & Credit Counseling Service", "end_date": "15 days", "taskType": 3, "help_text": "Add & tag local Credit & Debt Counseling Service (Organization)", "resources": ["14579", "14580"], "start_date": "today", "actionOrder": "2", "recur_every": "", "resource_reference_id": "14579, 14580"}, {"name": "[Schedule Credit & Debt Counseling as SGE or RGE]", "end_date": "15 days", "taskType": 3, "help_text": "", "resources": [], "start_date": "today", "actionOrder": "3", "recur_every": "", "resource_reference_id": ""}, {"name": "Rate resource for benefit of those who follow", "end_date": "15 days", "taskType": 3, "help_text": "", "resources": [], "start_date": "today", "actionOrder": "4", "recur_every": "", "resource_reference_id": ""}], "sequence_num": "1"}}
We use JSON_EXTRACT to extract a key from the JSON_COLUMN using the following syntax:
JSON_EXTRACT(json_field, '$.key')
If, however, we need to extract nested keys like in your case, we can either append the nested child keys to the path like
JSON_EXTRACT('{"resolve_housing_issues": {"tasks": [{"name": "Decision Map", "end_date": "15 days"}]}}', '$.resolve_housing_issues.tasks[0].name')
as is depicted in #bill-karwin's answer or make use of the wildcards like the following:
SELECT JSON_EXTRACT('{"resolve_housing_issues": {"tasks": [{"name": "Decision Map", "end_date": "15 days"}]}}', '$**.resolve_housing_issues') as resolve_housing_issues;
It produces the following result:
While the query
SELECT JSON_EXTRACT('{"resolve_housing_issues": {"tasks": [{"name": "Decision Map", "end_date": "15 days"}]}}', '$**.tasks') as tasks; produces the following result:
So on and so forth.
More on this can be found here.
The selector I was looking for seems to be a "$**" wildcard. This allowed me to grab all of the unnamed objects for each of the rows and the values of the nested resources.
JSON_EXTRACT(template_data, '$**.tasks[*].resources') AS resources
mysql> select json_extract(template_data, '$."Resolve housing issues".tasks[*].resources') as resources from goal_templates;
+----------------------------------+
| resources |
+----------------------------------+
| [[], ["14579", "14580"], [], []] |
+----------------------------------+

How can I see logs of the JSON post bodies sent by zapier to my CRM (Current RMS) via the Webhook zap during setup and testing?

I'm trying to send new users / new customres of my WooCommerce store into the rental management app current-rms.com as new Organisations / new contacts. Since Current RMS does not have a native Zap, I am trying to use the generic Webhook zap that Zapier maintains.
Specifically, I'd like to see the sent JSON body in Zapier posts that I make during the setup and testing of the Zap after clicking "Make a Zap!". The Task History is not detailed enough nor does it show hits during test and setup, since it's not live yet.
My trigger is a WooCommerce New Customer. This is working with Zapier WooCommerce Plugin and webhooks OK.
My action is the generic Zapier "Webhooks" Zap. The label "instant" appears next to it in the list at /app/zaps and it is "off".
One version uses JSON PAYLOAD as the action.
Another version uses CUSTOM PAYLOAD as the action.
Wrap request in array is YES.
Unflatten is YES.
My API key and subdomain are in the app URL as query strings and working OK.
When I hit test I get:
We had trouble sending your test through.
The app returned "Invalid JSON - missing or invalid entry for 'member'". This usually happens when your Zap is missing a required field or a field value isn't in a recognized format.
We made a request to api.current-rms.com and received (400) Bad Request.
Official docs are at: https://api.current-rms.com/doc#members-members-post
Logging available at Current RMS side
Part of the authentication of Current RMS involves knowing the domain of the account you are trying to access, in my case its therockfactory due to it being an account for my company https://therockfactory.net/
https://api.current-rms.com/api/v1/members?apikey=APIKEYCENSORED&subdomain=therockfactory
which returns the following when I use the correct API key:
{"webhook_logs":[],"meta":{"total_row_count":0,"row_count":0,"page":1,"per_page":20}}
Maybe if I could see the actual hit that Zapier is posting to Current I could wrap my confused brain around it better? What me worry.
The hit should look somewhat similar to this example, but I've not been able to locate it so far... (in Zapier)
Headers
Content-Type: application/json
Body
{
"member": {
"name": "Chris Bralton",
"description": "Pictures and leaned back was strewn at one would rather more. People don't want of his own means of one hand! Unless it from our pioneer has he fallen tree but that ever stronger and a. Hid among us against the full of verdure through by my eyes.",
"active": true,
"bookable": false,
"location_type": 0,
"locale": "en-GB",
"membership_type": "Contact",
"lawful_basis_type_id": 10001,
"sale_tax_class_id": 1,
"purchase_tax_class_id": 1,
"tag_list": [
"[\"Red\", \"Blue\", \"Green\"]"
],
"custom_fields": {},
"membership": {},
"primary_address": {
"name": "Chris Branson",
"street": "16 The Triangle",
"postcode": "NG2 1AE",
"city": "Nottingham",
"county": "Nottinghamshire",
"country_id": "1",
"country_name": "United Kingdom",
"type_id": 3001,
"address_type_name": "Primary",
"created_at": "2015-06-29T10:00:00.000Z",
"updated_at": "2015-06-29T10:30:00.000Z"
},
"emails": [
{
"address": "abigail.parker#ggmail.co.uk",
"type_id": 4001,
"email_type_name": "Work",
"id": 1
}
],
"phones": [
{
"number": "+44 115 9793399",
"type_id": 6001,
"phone_type_name": "Work",
"id": 1
}
],
"links": [
{
"address": "www.facebook.com/profile.php?id=566828251",
"type_id": 5002,
"link_type_name": "Facebook",
"id": 1
}
],
"addresses": [
{
"name": "Chris Branson",
"street": "16 The Triangle",
"postcode": "NG2 1AE",
"city": "Nottingham",
"county": "Nottinghamshire",
"country_id": "1",
"country_name": "United Kingdom",
"type_id": 3002,
"address_type_name": "Billing",
"created_at": "2017-06-29T10:00:00.000Z",
"updated_at": "2017-06-29T10:30:00.000Z",
"id": 1
}
],
"service_stock_levels": [
{
"item_id": 10,
"store_id": 1,
"member_id": 1,
"asset_number": "Chris Bralton",
"serial_number": "",
"location": "",
"stock_type": 3,
"stock_category": 60,
"quantity_held": "1.0",
"quantity_allocated": "0.0",
"quantity_unavailable": "0.0",
"quantity_on_order": "0.0",
"starts_at": "",
"ends_at": "",
"icon": {
"iconable_id": 85,
"id": 1,
"image_file_name": "abigail.jpeg",
"url": "https://s3.amazonaws.com/current-rms-development/64a0ccd0-5fbd-012f-2201-60f847290680/icons/46/original/abigail.jpeg",
"thumb_url": "https://s3.amazonaws.com/current-rms-development/64a0ccd0-5fbd-012f-2201-60f847290680/icons/46/thumb/abigail.jpeg",
"created_at": "2015-06-29T10:00:00.000Z",
"updated_at": "2015-06-29T10:30:00.000Z",
"iconable_type": "StockLevel"
},
"custom_fields": {},
"id": 487,
"item_name": "Sound Engineer",
"store_name": "Nottingham",
"stock_type_name": "Service",
"stock_category_name": "Resource"
}
],
"day_cost": "",
"hour_cost": "",
"distance_cost": "",
"flat_rate_cost": "",
"icon": {
"image": ""
},
"child_members": [
{
"relatable_id": 317,
"relatable_type": "Member",
"related_id": 25,
"related_type": "Member"
}
],
"parent_members": [
{
"relatable_id": 317,
"relatable_type": "Member",
"related_id": 25,
"related_type": "Member"
}
]
}
}
UPDATE: After reading my chosen answer I was able to see what Zapier was sending:
[
{
"member[emails_attributes][0][address]": "test#test.co.nz",
"member[membership_type]": "Organisation",
"member[name]": "Testafari Testing"
}
]
You can send your webhook to a tool like this one to inspect the payloads that are being sent from anywhere on the internet: https://requestbin.com/
You can find more help in regards to using Webhooks by Zapier and other ideas on how you can troubleshoot issues stemming from its use: https://zapier.com/apps/webhook/help#inspect-the-requests

JSON String gets cutoff When emoji is in data [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 6 years ago.
Improve this question
I have a VB.Net Webservice that returns data is a JSON string. Some of the data returned might have an EMOJI in it. When I return data that does not have an EMOJI in it the data come back fine.
Like this -
[
{
"ImgID": "150",
"ImgDesc": "Added to your portfolio",
"PublicID": "c0lbquvzitd3igohfh7x",
"ImgURL": "test.com",
"LikeCount": "0",
"CommentCnt": "2",
"PickCount": "",
"ServerUpdateDate": "1\/17\/2016 3:13:21 PM",
"UserName": "Deathstarspa",
"ProfileImgID": "hqhoiohfckzldgpdem6k",
"Location": "Chester County, PA, USA",
"dltflg": "False",
"PortName": "2 0 1 6 | M E N S",
"Categories": "Blowouts"
},
{
"ImgID": "151",
"ImgDesc": "Stone cold",
"PublicID": "udecff1bi0ymhewp1qlv",
"ImgURL": "Test.com",
"LikeCount": "0",
"CommentCnt": "0",
"PickCount": "",
"ServerUpdateDate": "1\/17\/2016 3:17:26 PM",
"UserName": "Deathstarspa",
"ProfileImgID": "hqhoiohfckzldgpdem6k",
"Location": "Chester County, PA, USA",
"dltflg": "False",
"PortName": "2 0 1 6 | M E N S",
"Categories": "Test"
}
]
If the data has an emoji anywhere in the sting the string gets cutoff at the very end.
It does not matter where the emoji is in the data.
Like -
[
{
"ImgID": "161",
"ImgDesc": "Bed ?????",
"PublicID": "gnw6ooo04ls21kqq8doo",
"ImgURL": "test.com",
"LikeCount": "0",
"CommentCnt": "0",
"PickCount": "",
"ServerUpdateDate": "2\/11\/2016 8:18:03 PM",
"UserName": "Deathstarspa",
"ProfileImgID": "hqhoiohfckzldgpdem6k",
"Location": "Chester County, PA, USA",
"dltflg": "False",
"PortName": "Bed Pics",
"Categories": "Blowouts,Treatments"
},
{
"ImgID": "162",
"ImgDesc": "Bed ?????",
"PublicID": "lrdxpvwxoq5zxpkagamo",
"ImgURL": "test.com",
"LikeCount": "0",
"CommentCnt": "0",
"PickCount": "",
"ServerUpdateDate": "2\/11\/2016 8:18:05 PM",
"UserName": "Deathstarspa",
"ProfileImgID": "hqhoiohfckzldgpdem6k",
"Location": "Chester County, PA, USA",
"dltflg": "False",
"PortName": "Test Pics"
Although you cannot see the Emojis in the ImgDesc piece the ????? represent the emoji. So the record with ImgID 161 is complete, but the record with ImgID of 162 get cutoff at "Test Pics,". If I remove the emoji from the ImgDesc then the data come back fine.
I am working with VB.NET and I have the Newtonsoft.Json dll referenced.
Is there something I need to do so the emojis do not cause issues?
You will need to properly encode the emoji, per the JSON standard. See the related answer below:
how can I Deserialize emoji in json in C#