Microsoft Migration Database Tool Improper character error - json

I am attempting use https://azure.microsoft.com/en-us/documentation/articles/documentdb-import-data/#Prerequisites to upload a json file to Azure DocumentDB. As soon as the upload gets going, I get this error:
System.ArgumentException: The resource name presented contains invalid character '/'.
at Microsoft.Azure.Documents.Client.DocumentClient.ValidateResource(Resource resource)
at Microsoft.Azure.Documents.Client.DocumentClient.d__33.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
I cant seem to find an documentation on this or maybe i dont know where to look. I would like some guidance on what this means and how to fix it. My guess is that "/" is an invalid character somewhere in my document, perhaps at line 33.
This is the document up to line 33 if that is what it indicates.
[
{
"name":"Cabinet De Lespinasse",
"accountid":2953,
"address":null,
"city":null,
"country":null,
"createdAt":null,
"heading":null,
"headingid":null,
"latitude":null,
"longitude":null,
"objectId":null,
"phonenumber1":null,
"website":null,
"display":null,
"email":null,
"aboutUs":"Education: Graduated from the Faculty of Law, State University of Haiti, 1983, Types of Practice: Established in 1877, the law office is specialized in intellectual property law for trademark and patent, insurance and commercial matters. Does not handle collection, Languages: Creole, French, English",
"Verified":false,
"Brands":null,
"Products":null,
"Services":null,
"Facebook":null,
"Instagram":null,
"googlePlus":null,
"Twitter":null,
"LinkedIn":null,
"Youtube":null,
"Pinterest":null,
"paymentMethods":null,
"languagesSpoken":null
},
{//<--This is line 33//
"name":"Cabinet Villejoint",
"accountid":2952,
"address":null,
Update:
When I upload one object in the following format, the upload succeeds. I removed the date field per #larry's comment below.
{
"name": "Western Union Agence Belmart ",
"accountid": 2851,
"address": "Route de Tabarre,",
"city": "Port-au-Prince",
"country": "Haiti",
"heading": "Money Transfer Locations",
"headingid": 428,
"latitude": null,
"longitude": null,
"objectId": "0NM7Bav1Lp",
"phonenumber1": "+509 3859 6346",
"website": "http://868.ht",
"display": "Western Union Agence Belmart",
"email": null,
"aboutUs": null,
"Verified": false,
"Brands": null,
"Products": null,
"Services": null,
"Facebook": null,
"Instagram": null,
"googlePlus": null,
"Twitter": null,
"LinkedIn": null,
"Youtube": null,
"Pinterest": null,
"paymentMethods": null,
"languagesSpoken": null
}
However, when I had a second object like below the upload fails. Any idea what is going on here?
[{
"name":"Western Union Agence Belmart ",
"accountid":2851,
"address":"Route de Tabarre,",
"city":"Port-au-Prince",
"country":"Haiti",
"heading":"Money Transfer Locations",
"headingid":428,
"latitude":null,
"longitude":null,
"objectId":"0NM7Bav1Lp",
"phonenumber1":"+509 3859 6346",
"website":"http://868.ht",
"display":"Western Union Agence Belmart",
"email":null,
"aboutUs":null,
"Verified":false,
"Brands":null,
"Products":null,
"Services":null,
"Facebook":null,
"Instagram":null,
"googlePlus":null,
"Twitter":null,
"LinkedIn":null,
"Youtube":null,
"Pinterest":null,
"paymentMethods":null,
"languagesSpoken":null
},
{
"name":"Western Union Agence Belmart ",
"accountid":2853,
"address":"Route de Tabarre,",
"city":"Port-au-Prince",
"country":"Haiti",
"heading":"Money Transfer Locations",
"headingid":428,
"latitude":null,
"longitude":null,
"objectId":"0NM7Bav1Lp",
"phonenumber1":"+509 3859 6346",
"website":"http://868.ht",
"display":"Western Union Agence Belmart",
"email":null,
"aboutUs":null,
"Verified":false,
"Brands":null,
"Products":null,
"Services":null,
"Facebook":null,
"Instagram":null,
"googlePlus":null,
"Twitter":null,
"LinkedIn":null,
"Youtube":null,
"Pinterest":null,
"paymentMethods":null,
"languagesSpoken":null
}]
Thank you.

Apparently the correct answer was provided by #larry maccherone below in that DocumentDb's Document Explorer in the Azure Portal only lets you upload one json object at a time. I missed this in the documentation or it doesnt say it there. That is, however, the result I experienced. Thanks, Larry.

Related

BIM360 Project's Service Type undocumented

On GET projects from BIM360, service_typesproperty contains services no documented.
Making a GET request to BIM360 Projects (https://forge.autodesk.com/en/docs/bim360/v1/reference/http/projects-:project_id-GET/), the service_types returns additional values than those documented on https://forge.autodesk.com/en/docs/bim360/v1/overview/parameters/#service-type
{
"id": "{PROJECT_ID}",
"account_id": "{ACCOUNT_ID}",
"name": "Sample Project",
"start_date": "2019-09-25",
"end_date": "2019-10-25",
"value": null,
"currency": null,
"status": "active",
"job_number": null,
"address_line_1": null,
"address_line_2": null,
"city": null,
"state_or_province": null,
"postal_code": null,
"country": "United States",
"business_unit_id": null,
"created_at": "2019-09-25T14:26:04.092Z",
"updated_at": "2019-10-04T16:44:25.271Z",
"project_type": "Demonstration Project",
"timezone": null,
"language": "en",
"construction_type": null,
"contract_type": null,
"last_sign_in": "2019-10-04T16:44:25.000Z",
"service_types": "doc_manager,insight,admin"
}
According to the documentation, only field, glue, schedule, plan and doc_managerare expected.
So at the moment the situation is that those services are not properly documented since they are not fully supported yet. They are actively working in making this services better and documented properly. There is an improvement process being worked on this quarter. Check back with us in a near future, a good structure and improvement to the bim360 service is coming. thank you for reaching out.

How do you get the first element out of a JSON String without knowing the name of the element in FileMaker 16 or 17?

I had an issue today with Filemaker on how to get the first element out of a json result without knowing the key.
Example $json result from an API call
{
"26298070": {
"task_id": "26298070",
"parent_id": "0",
"name": "DEPOT-0045 Research ODBC Model Extraction via Django To cut down on development time from Filemaker to Postgres",
"external_task_id": "32c8fd51-2066-42b9-b88b-8a2275fafc3f",
"external_parent_id": "64e7c829-d88e-48ae-9ba4-bb7a3871a7ce",
"level": "1",
"add_date": "2018-06-04 21:45:16",
"archived": "0",
"color": "#34C644",
"tags": "DEPOT-0045",
"budgeted": "1",
"checked_date": null,
"root_group_id": "91456",
"assigned_to": null,
"assigned_by": null,
"due_date": null,
"note": "",
"context": null,
"folder": null,
"repeat": null,
"billable": "0",
"budget_unit": "hours",
"public_hash": null,
"modify_time": null
}
}
I tried JSONGetElement( $json, "") and got the original json.
I tried JSONGetElement( $json, ".") and got the original json.
I tried JSONGetElement( $json, 1 ) and got nothing.
How do you get the first element out of a JSON String without knowing the name of the element in FileMaker 16 or 17?
Try this for the root element:
JSONListKeys ( $json ; "" )
result: 26298070
Once you get the root, you can get the child keys.
I remembered that FileMaker has a function to extract words from text so I thought I'd see what happened if I extracted the first word as a key.
I tried
JSONGetElement ( $json ; MiddleWords ( $json,1,1 ) )
and got the result I was looking for.
{
"add_date": "2018-06-04 21:45:16",
"archived": "0",
"assigned_by": null,
"assigned_to": null,
"billable": "0",
"budget_unit": "hours",
"budgeted": "1",
"checked_date": null,
"color": "#34C644",
"context": null,
"due_date": null,
"external_parent_id": "64e7c829-d88e-48ae-9ba4-bb7a3871a7ce",
"external_task_id": "32c8fd51-2066-42b9-b88b-8a2275fafc3f",
"folder": null,
"level": "1",
"modify_time": null,
"name": "DEPOT-0045 Research ODBC Model Extraction via Django To cut down on development time from Filemaker to Postgres",
"note": "",
"parent_id": "0",
"public_hash": null,
"repeat": null,
"root_group_id": "91456",
"tags": "DEPOT-0045",
"task_id": "26298070"
}
which makes it easy to parse simple JSON schema's that use attributes for keys.

Loading JSON data with apostrophe (single quote) i

This is specifically regarding insert of data into CrateDB,
I have a table with one column of type OBJECT. I am trying to insert JSON data into that using python and it is going along fine.
create table users (userdata OBJECT);
However, my data has apostrophe's in it i.e. single quotes and so the insert is failing. I have replicated the problem on the console:
The problem in the data below is with the "snr" field. I have tried putting in a backslash but that does not help.
Can someone tell me how I can get the following insert to
INSERT into users (userdata)
VALUES ('{"area_code": "2", "companyname": "TEC", "cos": "National24Hrs-Standard",
"country_code": "AUS", "cucm_dn": "26902", "ddi": "84236902", "department": null,
"device_type": "Cisco 8945 SIP", "divisionname": "Demonstrations",
"emailaddress": "bart.simpson#core.demo.telstra.com", "extension": "26902",
"extra1": null, "extra2": null, "extra3": null, "extra4": null,
"featuregroup": "Mobile Worker", "firstname": "Bart", "information": null,
"ippbxchosen": "TEC-CL-1", "language": "English - United States", "lastname": "Simpson",
"locationname": "Sydney", "mask": "61284236902", "pickupgroup": null, "postcode": "2000",
"presence": "Y", "role": "enduser", "security_profile": "EndUser-SP1",
"snr": "Bart\'s Mobile:0457503561,Bart Simpson:457503561",
"username": "bart.simpson#core.demo.telstra.com", "voicemail": "UCX-Advanced-CoS"}')
Inserting a JSON string into a CrateDB object typed column will always fail, this has afaik nothing to do with wrong escaping in your case. Either insert this as an CrateDB object (which I guess is what you want) to gain the benefits of accessing/searching each key/value, or change the data type to string.
See documentation about using object literals https://crate.io/docs/crate/reference/en/latest/general/ddl/data-types.html#object-literals.
Also most CrateDB client drivers do support binding data as objects.

Parsing JSON in SQL Server 2017 (Clearbit API call)

I'm pulling some data into a database on my local server with API calls via Clearbit provider. Everything was OK regarding parsing the data with SQL Server 2017 until I hit a bump.
I will go straight on the example for easier understanding.
This is the example of an API call output in JSON
{
"id": "384dfe0d-5bba-445e-a390-2d946dc84a12",
"name": "Honeywell",
"legalName": "Honeywell International Inc",
"domain": "honeywell.com",
"domainAliases": [
"honeywell.at",
"honeywell.it",
"evohome.info",
"wifithermostat.com",
"emsaviation.com",
"mytotalconnect.com",
"honeywell.nl",
"honeywell.co.za",
"honeywell.com.au",
"honeywell.ca",
"alliedsignal.com",
"emsdss.com",
"primusepic.com",
"alarmnet-me.com",
"lebow.com",
"honeywell.ie",
"honeywell.jp",
"honeywell.com.br",
"trendcontrol.co.uk",
"honeywellforjaguar.co.uk",
"aviaso.com",
"skyforce.co.uk",
"newenglandinstruments.com",
"honeywell.fi",
"alarmnet.com",
"skyconnect.com",
"skyforceuk.com",
"securitex.com",
"missionready.com",
"honeywellaerospace.com",
"formation.com",
"aclon.com",
"electrocorp.com",
"ultrak.com",
"satcom1.com",
"hsmpats.com",
"myaerospace.com",
"emsglobaltracking.com",
"fascocontrols.com",
"honeywellnow.com",
"bendixbrakes.com",
"elmwoodsensors.com",
"ovationselect.com",
"honeywellbusinessaviation.com",
"iflyaspire.com",
"btrinc.com",
"honeywellspecialtymaterials.com",
"magneticsensors.com",
"activeye.com",
"egarrett.com",
"novar-eds.com",
"aviaso.co.uk",
"chadwick-helmuth.com",
"datainstruments.com",
"lebowproducts.com",
"honeywell-produktkatalog.de",
"honeywellforjaguar.com",
"hobbs-corp.com",
"emsgt.com",
"honeywellaes.com",
"honeywellbuildingsolutions.com",
"satcom1.aero",
"honeywell-building-solutions.de",
"lifesafetydistribution.com",
"godirect.com",
"garrettbulletin.com",
"yourhomeexpert.com",
"aerospacetrading.com",
"sensorsystems.com",
"wifithermostat.info",
"honeywell-fachseminare.de",
"hobbscorporation.com",
"kcl.hu",
"honeywell.sk",
"esser.info",
"inertialsensor.com",
"sensotec.com",
"notifier.com",
"honeywellgreer.com",
"smartact.de",
"honeywellfire.com",
"iris-systems.com",
"honeywell.ru",
"lxei.com",
"thermalswitch.com",
"hightempsolutions.com",
"aubetech.com",
"honeywell-haustechnik.de",
"careersathoneywell.com",
"garrettbyhoneywell.com",
"honeywell.in",
"honeywell.cn",
"honeywell.com.mx",
"kcp.com",
"satamatics.com",
"myflite.com"
],
"site": {
"title": "Honeywell",
"h1": null,
"metaDescription": " We are blending products with software solutions to link people and businesses to the information they need to be more efficient, safer and connected. ",
"metaAuthor": null,
"phoneNumbers": [
"+1 877-271-8620",
"+1 800-633-3991",
"+1 877-841-2840",
"+1 480-353-3020",
"+1 973-455-3388",
"+1 973-204-9621",
"+32 2 728 20 45",
"+32 476 20 90 19",
"+44 7794 007289",
"+86 21 2219 6509"
],
"emailAddresses": [
"domains#honeywell.com",
"HoneywellPrivacy#honeywell.com",
"rob.ferris#honeywell.com",
"ilse.schouteden#honeywell.com",
"chris.martin2#honeywell.com",
"Anahi.Espinosa#honeywell.com",
"lydia.lu#honeywell.com",
"madhavi.jha#Honeywell.com",
"Steven.Brecken#Honeywell.com",
"Steve.Brecken#Honeywell.com",
"Eugene.Tan#Honeywell.com"
]
},
"category": {
"sector": "Consumer Discretionary",
"industryGroup": "Automobiles & Components",
"industry": "Automotive",
"subIndustry": "Automotive",
"sicCode": "3714",
"naicsCode": null
},
"tags": [
"Automotive",
"Enterprise",
"B2B",
"Electrical"
],
"description": " We are blending products with software solutions to link people and businesses to the information they need to be more efficient, safer and connected. ",
"foundedYear": 1936,
"location": "115 Tabor Rd, Morris Plains, NJ 07950, USA",
"timeZone": "America/New_York",
"utcOffset": -4,
"geo": {
"streetNumber": "115",
"streetName": "Tabor Road",
"subPremise": null,
"city": "Morris Plains",
"postalCode": "07950",
"state": "New Jersey",
"stateCode": "NJ",
"country": "United States",
"countryCode": "US",
"lat": 40.8358456,
"lng": -74.4771042
},
"logo": "https://logo.clearbit.com/honeywell.com",
"facebook": {
"handle": "293855263965203",
"likes": null
},
"linkedin": {
"handle": "company/honeywell"
},
"twitter": {
"handle": "HoneywellNow",
"id": "257492733",
"bio": "Please visit us over at #Honeywell.",
"followers": 2322,
"following": 1,
"location": "Morris Plains, NJ",
"site": "https:",
"avatar":
},
"crunchbase": {
"handle": "organization/honeywell"
},
"emailProvider": false,
"type": "public",
"ticker": "HON",
"phone": "+1 973-455-2000",
"metrics": {
"alexaUsRank": 6045,
"alexaGlobalRank": 18053,
"googleRank": null,
"employees": 51779,
"employeesRange": "1000+",
"marketCap": 102920000000,
"raised": null,
"annualRevenue": 39302000000,
"fiscalYearEnd": 12
},
"indexedAt": "2017-07-11T23:00:41.115Z",
"tech": [
"crazy_egg",
"google_analytics",
"google_tag_manager",
"asp_net",
"mouseflow",
"marketo",
"go_squared",
"microsoft_exchange_online",
"outlook",
"recaptcha"
],
"parent": {
"domain": null
},
"similarDomains": [
"abb-livingspace.com",
"alerton.com",
"gereports.com",
"honeywellprocess.com",
"honeywelluk.com",
"johnsoncontrols.com",
"jpinstruments.com",
"lenel.com",
"maxitrol.com",
"nucalgon.com",
"schneider-electric.us",
"siemens.com"
]
}
If you look at the example up here you will see "domainAliases": [...]
and that is the part of the JSON I still need to parse.
This is the parse query for SQL that I already have:
SELECT *
, JSON_VALUE(JSONData,'$.name') AS CompanyName
, JSON_VALUE(JSONData,'$.category.sector') AS CategorySector
, JSON_VALUE(JSONData, '$.category.industryGroup') AS CategoryIndustryGroup
, JSON_VALUE(JSONData, '$.category.industry') AS CategoryIndustry
, JSON_VALUE(JSONData, '$.category.subIndustry') AS CategorySubIndustry
, JSON_VALUE(JSONData, '$.category.sicCode') AS CategorySicCode
, JSON_VALUE(JSONData, '$.category.naicsCode') AS CategoryNaicsCode
, JSON_VALUE(JSONData, '$.metrics.employees') AS EmployeesNumber
, JSON_VALUE(JSONData, '$.metrics.employeesRange') AS EmployeesRange
, JSON_VALUE(JSONData, '$.metrics.marketCap') AS MarketCap
, JSON_VALUE(JSONData, '$.metrics.annualRevenue') AS AnnualRevenue
, JSON_VALUE(JSONData, '$.similarDomains') AS SimilarDomains
FROM Domains;
I want this data ("domainAliases") to be stored in other table as the data in the upper query (I know that the parse query I already have is only a SELECT query but I also have an UPDATE version of the query).
Here is an example picture of how the finished product in a new table, same database should look. The left column is called Company Name, the 2nd column is called Domain Aliases:
Now WHERE is the JSON data stored? I have it stored in a Column called JSONData, tablename: Domains and all this is in a database called Domainbank. JSONData datatype is nvarchar(max).
I need the data to be grouped by the name of the company and next to the company name there should be aliases domain just like the picture example shows. Now keep in mind that I will run this query for 10k+ JSONDatas and the new table that is going to be created will be super huge but as long as it is all grouped by the company name with all the alias domains it should be good. Some of the JSONDatas did not return the API call in the correct format because they either didn't find the data or something else went wrong, so If the query doesnt find anyting under the "domainAliases": [...] or if it doesn't even find the "domainAliases": [...] then I don't need the company to appear on the new table.
So short recap: let's make a new table (Let's call it AliasDomains), find the data under "domainAliases": [...] also pull the company name out JSON_VALUE(JSONData,'$.name') AS CompanyName, Store the data in the new table as the picture example higher in the post and then group by CompanyName.
So, from your post I am not completely clear on what your question is, but I assume it is how to write some SQL statement to accomplish the above?
First of all, I'd say you should not care of the GROUP BY in the insert, do GROUP BY when retrieving data out of the table.
Having said that you can quite easily accomplish what you want with a SELECT from the Domains table together with a CROSS APPLY OPENJSON statement, like so:
INSERT INTO AliasDomains(CompanyName, DomainAliases)
SELECT JSON_VALUE(JSONData, '$.name'), value
FROM Domains
CROSS APPLY OPENJSON (JSONData, '$.domainAliases')
EDIT: Should probably add that value in the above statement is returned from OPENJSON, e.g. it references the values of the (in this case domainAliases) path you want.
Hope this helps?!
Niels

JSON to CSV: How to add filters (columns) in the final Excel table?

First, I apologize if my description is not accurate enough for you, I am a total newbie and I don't know a thing about programming, so don't hesitate to tell me if you need more detailed info, but I will try to be as precise as possible.
So I have downloaded a bunch of tweets thanks to Twitter's API and the Terminal (through Twurl). All the tweets are in a .json file (that I open with TextWrangler, I'm on a Mac) and the thing is that when I export my .json file to a .csv file in order to process and analyze the data more easily thanks to Excel (or at least the Excel version of LibreOffice), I don't have all the parameters I would require for my study, I lack the "bio" part of each Tweet info present in the .json file. In other words, in my final table I have a column for the tweet ID, one for the tweet author, one for the text of the tweet itself and so on... But I don't have a column for the bio of the tweet author, whereas this information is displayed in the .json file itself. So my question is: is there a code or anything which would enable me to have one more column displaying some more info present in the basic .json file in my final .csv table?
Again, this may not be clear, so don't hesitate to tell me if you need me to highlight a specific point.
Thanks in advance for any insight, I really need help on this one, this is for a research project I need to carry on for my PhD, so any help would be more than welcome!
EDIT: As an example, here is a sample of the data I have for one tweet in my original .json file:
{
"created_at": "Mon Apr 28 09:00:40 +0000 2014",
"id": 460705144846712800,
"id_str": "460705144846712832",
"text": "Work can suck a dick today",
"source": "Twitter for iPhone",
"truncated": false,
"in_reply_to_status_id": null,
"in_reply_to_status_id_str": null,
"in_reply_to_user_id": null,
"in_reply_to_user_id_str": null,
"in_reply_to_screen_name": null,
"user": {
"id": 253350311,
"id_str": "253350311",
"name": "JEEEZUS",
"screen_name": "Maxi_Flex",
"location": "Southchestershire",
"url": "http://www.soundcloud.com/maxi_flex",
"description": "Jazz Personality.G Mentality.",
"protected": false,
"followers_count": 457,
"friends_count": 400,
"listed_count": 1,
"created_at": "Thu Feb 17 02:08:57 +0000 2011",
"favourites_count": 1229,
"utc_offset": null,
"time_zone": null,
"geo_enabled": true,
"verified": false,
"statuses_count": 13661,
"lang": "en",
"contributors_enabled": false,
"is_translator": false,
"is_translation_enabled": false,
"profile_background_color": "08ABFC",
"profile_background_image_url": "http://pbs.twimg.com/profile_background_images/444297891977244672/Z1BkfCFB.jpeg",
"profile_background_image_url_https": "https://pbs.twimg.com/profile_background_images/444297891977244672/Z1BkfCFB.jpeg",
"profile_background_tile": true,
"profile_image_url": "http://pbs.twimg.com/profile_images/454073282778902529/gCGicDBH_normal.jpeg",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/454073282778902529/gCGicDBH_normal.jpeg",
"profile_banner_url": "https://pbs.twimg.com/profile_banners/253350311/1392339276",
"profile_link_color": "FA05F2",
"profile_sidebar_border_color": "FFFFFF",
"profile_sidebar_fill_color": "DDEEF6",
"profile_text_color": "333333",
"profile_use_background_image": true,
"default_profile": false,
"default_profile_image": false,
"following": null,
"follow_request_sent": null,
"notifications": null
},
"geo": null,
"coordinates": null,
"place": null,
"contributors": null,
"retweet_count": 0,
"favorite_count": 0,
"entities": {
"hashtags": [],
"symbols": [],
"urls": [],
"user_mentions": []
},
"favorited": false,
"retweeted": false,
"filter_level": "medium",
"lang": "en"
}
So in the final csv file, I have some of the info I mentionned above, but what I would need to add in the csv file is the "description" part (bold) of each string. Any help would be appreciated!
The problem is probably that JSON is hierarchical and CSV is not. I'm guessing that you are only getting the top level JSON elements and not the nested objects. For example if your JSON is:
{
'name': 'test',
'author': {
'id': 123,
'created': ''
}
}
you are only getting 'name' and not 'author.id'? If this is the case, check out other questions on SO related to flattening JSON out for CSV e.g. flattening json to csv format
Any good JSON to CSV converter will work, try this one. If there is somehting funky in the JSON we need an example of the input JSON and what is getting spit out.
If you just need that one field enter the following command on the command line:
cat test.json | sed -n 's/.*description\":\"\([^"]*\)\".*/Description, \1/p' > result.csv
Where test.json is the file with all the JSON entries in it.
Here is the output from an example I ran:
cat test.json | sed -n 's/.*description\":\"\([^"]*\)\".*/\1/p'
Jazz Personality.G Mentality.
Jazz Personality.G Mentality.
Jazz Personality.G Mentality.
Jazz Personality.G Mentality.
If the file is very large you may need to split in to parts:
split -l N test.json part
Where N is the number of lines per part.