Twilio Event Streams Schema to Hive DDL - json

Question

I'm working with Twilio's Event Streams product. I'm successfully receiving events, parsing to models with Pydantic and all of this works.
Now I need to retain these historically. They're going to s3 and I plan on using Athena to query this data.
Here's the question -- the JSON schema for CallSummary events is nested and complex. Before I tediously create a CREATE TABLE statement for this model, is there an easier approach I should be following?
For reference, I'm building the Python models using datamodel-codegen (link) like this:
curl -s https://events-schemas.twilio.com/VoiceInsights.CallSummary/1 > call_summary_event.json
datamodel-codegen --input call_summary_event.json --input-file-type jsonschema --output call_summary_model.py
Using 1 call summary record and hive-json-schema (from quux00), I can get about half of the fields:
java -cp target/json-hive-schema-1.0.jar net.thornydev.JsonHiveSchema CA0000123.json
CREATE TABLE x (
metrics array<struct<account_sid:string, call_sid:string, carrier_edge:null, client_edge:null, direction:string, edge:string, sdk_edge:null, sip_edge:struct<codec:int, codec_name:string, cumulative:struct<jitter:struct<avg:double, max:double>, packets_lost:int, packets_sent:int>, interval:struct<jitter:struct<avg:double, max:double>, packets_loss_percentage:double, packets_lost:int, packets_sent:int>, metadata:struct<edge_location:string, region:string, twilio_ip:string>>, timestamp:string>>)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe';
Unfortunately, much of the data in my call summary json records are not populated, so I can't easily use a script like this to convert the sample data to CREATE TABLE with nested structs.
Before I go about doing this further, is there something I could use to go from the published JSON schema straight to Hive DDL?

Answer 1

Using Glue Crawler, I was able to get this data created. Note that objects are stored in s3 paths with year/month/date partitions e.g. s3://bucket/com.twilio.voice.insights.call-summary.complete/year=2022/month=10/day=10//CA000123.json
'Create Table' DDL statement from Athena after table was created by Crawler:
CREATE EXTERNAL TABLE `com_twilio_voice_insights_call_summary_complete`(
`specverson` string COMMENT 'from deserializer',
`type` string COMMENT 'from deserializer',
`source` string COMMENT 'from deserializer',
`id` string COMMENT 'from deserializer',
`dataschema` string COMMENT 'from deserializer',
`datacontenttype` string COMMENT 'from deserializer',
`time` string COMMENT 'from deserializer',
`data` struct<call_sid:string,account_sid:string,parent_call_sid:string,parent_account_sid:string,start_time:string,end_time:string,duration:int,connect_duration:int,call_type:string,call_state:string,from_:struct<caller:string,callee:string,carrier:string,connection:string,number_prefix:string,location:struct<lat:double,lon:double>,city:string,country_code:string,country_subdivision:string,ip_address:string,sdk:struct<type:string,version:string,platform:string,region:string,selected_region:string,browser:struct<name:string,major:string,version:string>,os:struct<name:string,version:string>,device:struct<model:string,type:string,vendor:string,arch:string>,engine:struct<name:string,version:string>>>,to:struct<caller:string,callee:string,carrier:string,connection:string,number_prefix:string,location:struct<lat:double,lon:double>,city:string,country_code:string,country_subdivision:string,ip_address:string,sdk:struct<type:string,version:string,platform:string,region:string,selected_region:string,browser:struct<name:string,major:string,version:string>,os:struct<name:string,version:string>,device:struct<model:string,type:string,vendor:string,arch:string>,engine:struct<name:string,version:string>>>,processing_state:string,processing_version:int,sip_edge:struct<properties:struct<q850_cause:int,last_sip_response_num:int,pdd_ms:int,route_id:string,media_region:string,signaling_region:string,twilio_media_ip:string,twilio_signaling_ip:string,external_media_ip:string,external_signaling_ip:string,sip_call_id:string,user_agent:string,selected_region:string,region:string,trunk_sid:string,disconnected_by:string,direction:string,settings:string>,metrics:struct<inbound:struct<codec:int,codec_name:string,packets_received:int,packets_sent:string,packets_lost:int,packets_loss_percentage:double,jitter:struct<min:double,max:double,avg:double,value:string>,rtt:string,mos:string,audio_in:string,audio_out:string,latency:string,bytes_received:string,bytes_sent:string,packet_delay_variation:string>,outbound:struct<codec:int,codec_name:string,packets_received:string,packets_sent:int,packets_lost:int,packets_loss_percentage:double,jitter:struct<min:double,max:double,avg:double,value:string>,rtt:string,mos:string,audio_in:string,audio_out:string,latency:struct<min:double,max:double,avg:double,value:string>,bytes_received:string,bytes_sent:string,packet_delay_variation:struct<d50:int,d70:int,d90:int,d120:int,d150:int,d200:int,d300:int>>>,tags:array<string>,events:string>,carrier_edge:struct<properties:struct<q850_cause:int,last_sip_response_num:int,pdd_ms:int,route_id:string,media_region:string,signaling_region:string,twilio_media_ip:string,twilio_signaling_ip:string,external_media_ip:string,external_signaling_ip:string,sip_call_id:string,user_agent:string,selected_region:string,region:string,trunk_sid:string,disconnected_by:string,direction:string,settings:string>,metrics:struct<inbound:struct<codec:int,codec_name:string,packets_received:int,packets_sent:string,packets_lost:int,packets_loss_percentage:double,jitter:struct<min:double,max:double,avg:double,value:string>,rtt:string,mos:string,audio_in:string,audio_out:string,latency:string,bytes_received:string,bytes_sent:string,packet_delay_variation:struct<d50:int,d70:int,d90:int,d120:int,d150:int,d200:int,d300:int>>,outbound:struct<codec:int,codec_name:string,packets_received:string,packets_sent:int,packets_lost:int,packets_loss_percentage:double,jitter:struct<min:double,max:double,avg:double,value:string>,rtt:string,mos:string,audio_in:string,audio_out:string,latency:struct<min:double,max:double,avg:double,value:string>,bytes_received:string,bytes_sent:string,packet_delay_variation:struct<d50:int,d70:int,d90:int,d120:int,d150:int,d200:int,d300:int>>>,tags:array<string>,events:string>,sdk_edge:struct<properties:struct<q850_cause:int,last_sip_response_num:int,pdd_ms:int,route_id:string,media_region:string,signaling_region:string,twilio_media_ip:string,twilio_signaling_ip:string,external_media_ip:string,external_signaling_ip:string,sip_call_id:string,user_agent:string,selected_region:string,region:string,trunk_sid:string,disconnected_by:string,direction:string,settings:struct<ice_restart_enabled:boolean,dscp:boolean,edge:string,selected_edges:array<string>>>,metrics:struct<inbound:struct<codec:string,codec_name:string,packets_received:int,packets_sent:string,packets_lost:int,packets_loss_percentage:double,jitter:struct<min:double,max:double,avg:double,value:string>,rtt:struct<min:double,max:double,avg:double,value:string>,mos:struct<min:double,max:double,avg:double,value:string>,audio_in:struct<min:double,max:double,avg:double,value:string>,audio_out:struct<min:double,max:double,avg:double,value:string>,latency:string,bytes_received:string,bytes_sent:string,packet_delay_variation:string>,outbound:struct<codec:int,codec_name:string,packets_received:string,packets_sent:int,packets_lost:string,packets_loss_percentage:string,jitter:string,rtt:string,mos:string,audio_in:string,audio_out:string,latency:string,bytes_received:string,bytes_sent:string,packet_delay_variation:string>>,tags:array<string>,events:struct<groups:struct<settings:int,network_information:int,pc_connection_state:int,dtls_transport_state:int,audio_level_warning_cleared:int,audio_level_warning_raised:int,ice_candidate:int,network_quality_warning_raised:int,signaling_state:int,connection:int,get_user_media:int,ice_connection_state:int,ice_gathering_state:int,network_quality_warning_cleared:int,audio:int>,levels:struct<info:int,warning:int,debug:int,error:int>,errors:struct<31201:int,31208:int,53405:int,31000:int,31003:int>,feedback:struct<reason:string,score:int>>>,client_edge:struct<properties:struct<q850_cause:int,last_sip_response_num:int,pdd_ms:int,route_id:string,media_region:string,signaling_region:string,twilio_media_ip:string,twilio_signaling_ip:string,external_media_ip:string,external_signaling_ip:string,sip_call_id:string,user_agent:string,selected_region:string,region:string,trunk_sid:string,disconnected_by:string,direction:string,settings:string>,metrics:struct<inbound:struct<codec:int,codec_name:string,packets_received:int,packets_sent:string,packets_lost:int,packets_loss_percentage:double,jitter:struct<min:double,max:double,avg:double,value:string>,rtt:string,mos:string,audio_in:string,audio_out:string,latency:string,bytes_received:string,bytes_sent:string,packet_delay_variation:struct<d50:int,d70:int,d90:int,d120:int,d150:int,d200:int,d300:int>>,outbound:struct<codec:int,codec_name:string,packets_received:string,packets_sent:int,packets_lost:int,packets_loss_percentage:double,jitter:struct<min:double,max:double,avg:double,value:string>,rtt:string,mos:string,audio_in:string,audio_out:string,latency:struct<min:double,max:double,avg:double,value:string>,bytes_received:string,bytes_sent:string,packet_delay_variation:struct<d50:int,d70:int,d90:int,d120:int,d150:int,d200:int,d300:int>>>,tags:array<string>,events:string>,tags:array<string>,attributes:struct<conference_participant:boolean>,properties:struct<q850_cause:int,last_sip_response_num:int,pdd_ms:int,route_id:string,media_region:string,signaling_region:string,twilio_media_ip:string,twilio_signaling_ip:string,external_media_ip:string,external_signaling_ip:string,sip_call_id:string,user_agent:string,selected_region:string,region:string,trunk_sid:string,disconnected_by:string,direction:string,settings:string>> COMMENT 'from deserializer')
PARTITIONED BY (
`year` string,
`month` string,
`day` string,
`day` string)
ROW FORMAT SERDE
'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
'paths'='data,datacontenttype,dataschema,id,source,specverson,time,type')
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://bucket/'
TBLPROPERTIES (
'CrawlerSchemaDeserializerVersion'='1.0',
'CrawlerSchemaSerializerVersion'='1.0',
'UPDATED_BY_CRAWLER'='com.twilio.voice.insights.call-summary.complete',
'averageRecordSize'='4048',
'classification'='json',
'compressionType'='none',
'objectCount'='1240',
'recordCount'='1239',
'sizeKey'='5030025',
'typeOfData'='file')