I'm trying to check if a value contains a partial string in this sample API call result in JSON format. It should look through job title text to see if it contains the word "Virtual" or "Remote" anywhere in the string. If so, append 1 to new dictionary, and if not, append 0. I keep getting 1's for each value, but not sure why 0's don't show up if it can't find any match. Thanks! Here's the code:
import requests
remote_job_check = {'Remote':[]}
job_lists = {
"jobs": [
{
"country_code": "US",
"state": "AL",
"title": "Sr. Customer Solutions Manager - Virtual Opportunities",
},
{
"country_code": "US",
"state": "CA",
"title": "Data Engineer",
},
{
"country_code": "US",
"state": "CA",
"title": "Data Engineer - Remote",
},
{
"country_code": "US",
"state": "NV",
"title": "Remote Software Engineer",
},
{
"country_code": "US",
"state": "FL",
"title": "Product Manager",
}
]
}
job_data = requests.get(job_lists).json()['jobs']
for details in job_data:
if 'Virtual' or 'Remote' in details.get('title'):
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
I was able to figure it out. Thanks for the suggestions though. I learned that my original code only searches for one of the strings. The 'or' operator wasn't even working in this case:
for details in job_data:
if 'Virtual' or 'Remote' in details.get('title'): # The 'or' operator not working
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
Then I tried using just one string and it worked.
for details in job_data:
if 'Virtual' in details.get('title'): # This works
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
So now I figured I need to create some sort of array to iterate through and found the 'any' function. Here's the working code:
remote_job_keywords = ['Virtual', 'Remote']
if any(keyword in details['title'] for keyword in remote_job_keywords):
job_data['Remote'].append(1)
else:
job_data['Remote'].append(0)
UPDATE: Thanks to #shriakhilc for the feedback on my answer. Like he said, the above works but it was easier to fix the or operator like so:
for details in job_data:
if 'Virtual' in details.get('title') or 'Remote' in details.get('title'):
remote_job_check['Remote'].append(1)
else:
remote_job_check['Remote'].append(0)
This is specifically regarding insert of data into CrateDB,
I have a table with one column of type OBJECT. I am trying to insert JSON data into that using python and it is going along fine.
create table users (userdata OBJECT);
However, my data has apostrophe's in it i.e. single quotes and so the insert is failing. I have replicated the problem on the console:
The problem in the data below is with the "snr" field. I have tried putting in a backslash but that does not help.
Can someone tell me how I can get the following insert to
INSERT into users (userdata)
VALUES ('{"area_code": "2", "companyname": "TEC", "cos": "National24Hrs-Standard",
"country_code": "AUS", "cucm_dn": "26902", "ddi": "84236902", "department": null,
"device_type": "Cisco 8945 SIP", "divisionname": "Demonstrations",
"emailaddress": "bart.simpson#core.demo.telstra.com", "extension": "26902",
"extra1": null, "extra2": null, "extra3": null, "extra4": null,
"featuregroup": "Mobile Worker", "firstname": "Bart", "information": null,
"ippbxchosen": "TEC-CL-1", "language": "English - United States", "lastname": "Simpson",
"locationname": "Sydney", "mask": "61284236902", "pickupgroup": null, "postcode": "2000",
"presence": "Y", "role": "enduser", "security_profile": "EndUser-SP1",
"snr": "Bart\'s Mobile:0457503561,Bart Simpson:457503561",
"username": "bart.simpson#core.demo.telstra.com", "voicemail": "UCX-Advanced-CoS"}')
Inserting a JSON string into a CrateDB object typed column will always fail, this has afaik nothing to do with wrong escaping in your case. Either insert this as an CrateDB object (which I guess is what you want) to gain the benefits of accessing/searching each key/value, or change the data type to string.
See documentation about using object literals https://crate.io/docs/crate/reference/en/latest/general/ddl/data-types.html#object-literals.
Also most CrateDB client drivers do support binding data as objects.
I'm pulling some data into a database on my local server with API calls via Clearbit provider. Everything was OK regarding parsing the data with SQL Server 2017 until I hit a bump.
I will go straight on the example for easier understanding.
This is the example of an API call output in JSON
{
"id": "384dfe0d-5bba-445e-a390-2d946dc84a12",
"name": "Honeywell",
"legalName": "Honeywell International Inc",
"domain": "honeywell.com",
"domainAliases": [
"honeywell.at",
"honeywell.it",
"evohome.info",
"wifithermostat.com",
"emsaviation.com",
"mytotalconnect.com",
"honeywell.nl",
"honeywell.co.za",
"honeywell.com.au",
"honeywell.ca",
"alliedsignal.com",
"emsdss.com",
"primusepic.com",
"alarmnet-me.com",
"lebow.com",
"honeywell.ie",
"honeywell.jp",
"honeywell.com.br",
"trendcontrol.co.uk",
"honeywellforjaguar.co.uk",
"aviaso.com",
"skyforce.co.uk",
"newenglandinstruments.com",
"honeywell.fi",
"alarmnet.com",
"skyconnect.com",
"skyforceuk.com",
"securitex.com",
"missionready.com",
"honeywellaerospace.com",
"formation.com",
"aclon.com",
"electrocorp.com",
"ultrak.com",
"satcom1.com",
"hsmpats.com",
"myaerospace.com",
"emsglobaltracking.com",
"fascocontrols.com",
"honeywellnow.com",
"bendixbrakes.com",
"elmwoodsensors.com",
"ovationselect.com",
"honeywellbusinessaviation.com",
"iflyaspire.com",
"btrinc.com",
"honeywellspecialtymaterials.com",
"magneticsensors.com",
"activeye.com",
"egarrett.com",
"novar-eds.com",
"aviaso.co.uk",
"chadwick-helmuth.com",
"datainstruments.com",
"lebowproducts.com",
"honeywell-produktkatalog.de",
"honeywellforjaguar.com",
"hobbs-corp.com",
"emsgt.com",
"honeywellaes.com",
"honeywellbuildingsolutions.com",
"satcom1.aero",
"honeywell-building-solutions.de",
"lifesafetydistribution.com",
"godirect.com",
"garrettbulletin.com",
"yourhomeexpert.com",
"aerospacetrading.com",
"sensorsystems.com",
"wifithermostat.info",
"honeywell-fachseminare.de",
"hobbscorporation.com",
"kcl.hu",
"honeywell.sk",
"esser.info",
"inertialsensor.com",
"sensotec.com",
"notifier.com",
"honeywellgreer.com",
"smartact.de",
"honeywellfire.com",
"iris-systems.com",
"honeywell.ru",
"lxei.com",
"thermalswitch.com",
"hightempsolutions.com",
"aubetech.com",
"honeywell-haustechnik.de",
"careersathoneywell.com",
"garrettbyhoneywell.com",
"honeywell.in",
"honeywell.cn",
"honeywell.com.mx",
"kcp.com",
"satamatics.com",
"myflite.com"
],
"site": {
"title": "Honeywell",
"h1": null,
"metaDescription": " We are blending products with software solutions to link people and businesses to the information they need to be more efficient, safer and connected. ",
"metaAuthor": null,
"phoneNumbers": [
"+1 877-271-8620",
"+1 800-633-3991",
"+1 877-841-2840",
"+1 480-353-3020",
"+1 973-455-3388",
"+1 973-204-9621",
"+32 2 728 20 45",
"+32 476 20 90 19",
"+44 7794 007289",
"+86 21 2219 6509"
],
"emailAddresses": [
"domains#honeywell.com",
"HoneywellPrivacy#honeywell.com",
"rob.ferris#honeywell.com",
"ilse.schouteden#honeywell.com",
"chris.martin2#honeywell.com",
"Anahi.Espinosa#honeywell.com",
"lydia.lu#honeywell.com",
"madhavi.jha#Honeywell.com",
"Steven.Brecken#Honeywell.com",
"Steve.Brecken#Honeywell.com",
"Eugene.Tan#Honeywell.com"
]
},
"category": {
"sector": "Consumer Discretionary",
"industryGroup": "Automobiles & Components",
"industry": "Automotive",
"subIndustry": "Automotive",
"sicCode": "3714",
"naicsCode": null
},
"tags": [
"Automotive",
"Enterprise",
"B2B",
"Electrical"
],
"description": " We are blending products with software solutions to link people and businesses to the information they need to be more efficient, safer and connected. ",
"foundedYear": 1936,
"location": "115 Tabor Rd, Morris Plains, NJ 07950, USA",
"timeZone": "America/New_York",
"utcOffset": -4,
"geo": {
"streetNumber": "115",
"streetName": "Tabor Road",
"subPremise": null,
"city": "Morris Plains",
"postalCode": "07950",
"state": "New Jersey",
"stateCode": "NJ",
"country": "United States",
"countryCode": "US",
"lat": 40.8358456,
"lng": -74.4771042
},
"logo": "https://logo.clearbit.com/honeywell.com",
"facebook": {
"handle": "293855263965203",
"likes": null
},
"linkedin": {
"handle": "company/honeywell"
},
"twitter": {
"handle": "HoneywellNow",
"id": "257492733",
"bio": "Please visit us over at #Honeywell.",
"followers": 2322,
"following": 1,
"location": "Morris Plains, NJ",
"site": "https:",
"avatar":
},
"crunchbase": {
"handle": "organization/honeywell"
},
"emailProvider": false,
"type": "public",
"ticker": "HON",
"phone": "+1 973-455-2000",
"metrics": {
"alexaUsRank": 6045,
"alexaGlobalRank": 18053,
"googleRank": null,
"employees": 51779,
"employeesRange": "1000+",
"marketCap": 102920000000,
"raised": null,
"annualRevenue": 39302000000,
"fiscalYearEnd": 12
},
"indexedAt": "2017-07-11T23:00:41.115Z",
"tech": [
"crazy_egg",
"google_analytics",
"google_tag_manager",
"asp_net",
"mouseflow",
"marketo",
"go_squared",
"microsoft_exchange_online",
"outlook",
"recaptcha"
],
"parent": {
"domain": null
},
"similarDomains": [
"abb-livingspace.com",
"alerton.com",
"gereports.com",
"honeywellprocess.com",
"honeywelluk.com",
"johnsoncontrols.com",
"jpinstruments.com",
"lenel.com",
"maxitrol.com",
"nucalgon.com",
"schneider-electric.us",
"siemens.com"
]
}
If you look at the example up here you will see "domainAliases": [...]
and that is the part of the JSON I still need to parse.
This is the parse query for SQL that I already have:
SELECT *
, JSON_VALUE(JSONData,'$.name') AS CompanyName
, JSON_VALUE(JSONData,'$.category.sector') AS CategorySector
, JSON_VALUE(JSONData, '$.category.industryGroup') AS CategoryIndustryGroup
, JSON_VALUE(JSONData, '$.category.industry') AS CategoryIndustry
, JSON_VALUE(JSONData, '$.category.subIndustry') AS CategorySubIndustry
, JSON_VALUE(JSONData, '$.category.sicCode') AS CategorySicCode
, JSON_VALUE(JSONData, '$.category.naicsCode') AS CategoryNaicsCode
, JSON_VALUE(JSONData, '$.metrics.employees') AS EmployeesNumber
, JSON_VALUE(JSONData, '$.metrics.employeesRange') AS EmployeesRange
, JSON_VALUE(JSONData, '$.metrics.marketCap') AS MarketCap
, JSON_VALUE(JSONData, '$.metrics.annualRevenue') AS AnnualRevenue
, JSON_VALUE(JSONData, '$.similarDomains') AS SimilarDomains
FROM Domains;
I want this data ("domainAliases") to be stored in other table as the data in the upper query (I know that the parse query I already have is only a SELECT query but I also have an UPDATE version of the query).
Here is an example picture of how the finished product in a new table, same database should look. The left column is called Company Name, the 2nd column is called Domain Aliases:
Now WHERE is the JSON data stored? I have it stored in a Column called JSONData, tablename: Domains and all this is in a database called Domainbank. JSONData datatype is nvarchar(max).
I need the data to be grouped by the name of the company and next to the company name there should be aliases domain just like the picture example shows. Now keep in mind that I will run this query for 10k+ JSONDatas and the new table that is going to be created will be super huge but as long as it is all grouped by the company name with all the alias domains it should be good. Some of the JSONDatas did not return the API call in the correct format because they either didn't find the data or something else went wrong, so If the query doesnt find anyting under the "domainAliases": [...] or if it doesn't even find the "domainAliases": [...] then I don't need the company to appear on the new table.
So short recap: let's make a new table (Let's call it AliasDomains), find the data under "domainAliases": [...] also pull the company name out JSON_VALUE(JSONData,'$.name') AS CompanyName, Store the data in the new table as the picture example higher in the post and then group by CompanyName.
So, from your post I am not completely clear on what your question is, but I assume it is how to write some SQL statement to accomplish the above?
First of all, I'd say you should not care of the GROUP BY in the insert, do GROUP BY when retrieving data out of the table.
Having said that you can quite easily accomplish what you want with a SELECT from the Domains table together with a CROSS APPLY OPENJSON statement, like so:
INSERT INTO AliasDomains(CompanyName, DomainAliases)
SELECT JSON_VALUE(JSONData, '$.name'), value
FROM Domains
CROSS APPLY OPENJSON (JSONData, '$.domainAliases')
EDIT: Should probably add that value in the above statement is returned from OPENJSON, e.g. it references the values of the (in this case domainAliases) path you want.
Hope this helps?!
Niels
I'm reading JSON data from an ARC Server report online and trying to create a database with the data.
I've created the database named: test.db
I need the columns to be identified as "Service", "Folder", "Service URL", "Configured State", "Real Time State", "Server Type".
and the rows as each "Service" returned from the report.
The JSON data looks like this:
{"reports": [{
"folderName": "/",
"serviceName": "SampleWorldCities",
"type": "MapServer",
"description": "The SampleWorldCities service is provided so you can quickly and easily preview the functionality of the GIS server. Click the thumbnail image to open in a web application. This sample service is optional and can be deleted.",
"isDefault": false,
"isPrivate": false,
"hasManifest": false,
"status": {
"configuredState": "STARTED",
"realTimeState": "STARTED"
},
"instances": {
"folderName": "/",
"serviceName": "SampleWorldCities",
"type": "MapServer",
"max": 1,
"busy": 0,
"free": 1,
"initializing": 0,
"notCreated": 0,
"transactions": 72,
"totalBusyTime": 127611,
"isStatisticsAvailable": true
},
"properties": {
"maxRecordCount": "1000",
"filePath": "${AGSSERVER}/framework/etc/data/WorldCities/WorldCities.msd",
"cacheOnDemand": "false",
"useLocalCacheDir": "true",
"outputDir": "/home/ec2-user/arcgis/server/usr/directories/arcgisoutput",
"virtualOutputDir": "/rest/directories/arcgisoutput",
"supportedImageReturnTypes": "MIME+URL",
"minScale": "295000000",
"isCached": "false",
"ignoreCache": "false",
"maxScale": "4000",
"clientCachingAllowed": "true",
"cacheDir": "/home/ec2-user/arcgis/server/usr/directories/arcgiscache"
},
"iteminfo": {
"description": "The SampleWorldCities service is provided so you can quickly and easily preview the functionality of the GIS server. Click the thumbnail image to open in a web application. This sample service is optional and can be deleted.",
"summary": "The SampleWorldCities service is provided so you can quickly and easily preview the functionality of the GIS server. Click the thumbnail image to open in a web application. This sample service is optional and can be deleted.",
"tags": [
"sample",
"map",
"service"
],
"thumbnail": "thumbnail.png"
},
"permissions": [{
"principal": "esriEveryone",
"permission": {"isAllowed": true},
"childURL": null,
"operation": null
}]
}]}
my sript is as follows:
import json
import sqlite3
db = sqlite3.connect('test.db')
traffic = json_read
c = db.cursor()
someitem = traffic.itervalues().next()
columns = ['Service', 'Folder', 'Service URL', 'Configured State', 'Real Time State', 'Server Type']
c.execute("SELECT sql FROM sqlite_master WHERE " \
"Service='Services' AND type = 'table'")
create_table_string = cursor.fetchall()[0][0]
c.execute('''create table Services
(Service text primary key,
Folder text,
Service URL text,
Configured State text,
Real Time State text,
Server Type text)''')
for service, data in traffic.iteritems():
services = (service,) + tuple(data[c] for c in columns)
c = db.cursor()
c.execute(query)
c.close()
print "JSON Complete"
Can someone point me in the right direction?
Forgot to mention
Service is Service Name,
Folder is folder name,
service url is link to the service,
configured state is configuredState,
realtime state is realTimeState,
Server type is type
db = sqlite3.connect(
'server.db')
cursor = db.cursor()
cursor.execute("DROP TABLE if exists Services")
db.commit()
cursor.execute("DROP TABLE if exists Services2")
db.commit()
cursor.execute('''CREATE TABLE Services
(Service text,
Folder text,
Service_URL text,
Configured_State text,
Real_Time_State text,
Server text);''')
This was the code that gave me the output I was looking for.