Exploding Entire JSON File in PySpark - json

Question

I am trying to normalize (perhaps not the precise term) a nested JSON object in PySpark. The actual data I care about is under articles. The schema is:
df = spark.read.json(filepath)
df.printSchema()
root
|-- articles: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- adultLanguage: string (nullable = true)
| | |-- companies: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- contentCount: long (nullable = true)
| | | | |-- exchange: string (nullable = true)
| | | | |-- isin: string (nullable = true)
| | | | |-- name: string (nullable = true)
| | | | |-- primary: boolean (nullable = true)
| | | | |-- symbol: string (nullable = true)
| | | | |-- titleCount: long (nullable = true)
| | |-- content: string (nullable = true)
| | |-- copyright: string (nullable = true)
| | |-- duplicateGroupId: string (nullable = true)
| | |-- estimatedPublishedDate: string (nullable = true)
| | |-- harvestDate: string (nullable = true)
| | |-- id: string (nullable = true)
| | |-- indexTerms: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- domains: array (nullable = true)
| | | | | |-- element: string (containsNull = true)
| | | | |-- name: string (nullable = true)
| | | | |-- score: string (nullable = true)
| | |-- language: string (nullable = true)
| | |-- languageCode: string (nullable = true)
| | |-- licenses: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- name: string (nullable = true)
| | |-- linkedArticles: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- originalUrl: string (nullable = true)
| | |-- print: struct (nullable = true)
| | | |-- pageNumber: string (nullable = true)
| | | |-- publicationEdition: string (nullable = true)
| | |-- publishedDate: string (nullable = true)
| | |-- publishingPlatform: struct (nullable = true)
| | | |-- itemId: string (nullable = true)
| | |-- semantics: struct (nullable = true)
| | | |-- entities: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- properties: array (nullable = true)
| | | | | | |-- element: struct (containsNull = true)
| | | | | | | |-- name: string (nullable = true)
| | | | | | | |-- value: string (nullable = true)
| | | | | |-- provider: string (nullable = true)
| | |-- sentiment: struct (nullable = true)
| | | |-- score: string (nullable = true)
| | |-- sequenceId: string (nullable = true)
| | |-- source: struct (nullable = true)
| | | |-- category: string (nullable = true)
| | | |-- editorialRank: string (nullable = true)
| | | |-- feed: struct (nullable = true)
| | | | |-- id: string (nullable = true)
| | | | |-- idFromPublisher: string (nullable = true)
| | | | |-- inWhiteList: string (nullable = true)
| | | | |-- language: string (nullable = true)
| | | | |-- mediaType: string (nullable = true)
| | | | |-- publishingPlatform: string (nullable = true)
| | | | |-- rank: struct (nullable = true)
| | | | | |-- inboundLinkCount: string (nullable = true)
| | | |-- homeUrl: string (nullable = true)
| | | |-- id: string (nullable = true)
| | | |-- location: struct (nullable = true)
| | | | |-- country: string (nullable = true)
| | | | |-- countryCode: string (nullable = true)
| | | | |-- region: string (nullable = true)
| | | | |-- state: string (nullable = true)
| | | | |-- subregion: string (nullable = true)
| | | |-- name: string (nullable = true)
| | |-- title: string (nullable = true)
| | |-- topics: array (nullable = true)
| | | |-- element: struct (containsNull = true)
| | | | |-- name: string (nullable = true)
| | |-- url: string (nullable = true)
| | |-- wordCount: string (nullable = true)
|-- status: string (nullable = true)
|-- totalResults: string (nullable = true)
I can successfully use select to grab variables that are strings, such as:
df1 = df.select(explode(df.articles).alias('articles'))
df2 = df1.select(
'articles.adultLanguage',
...)
but I don't know how I can get variables that are multiple levels down, such as author and companies that are arrays with potentially multiple values.
{"status": "SUCCESS", "totalResults": "294629", "articles": [{"sequenceId": "557545999680", "id": "24414529721", "language": "English", "languageCode": "en", "title": "Young CEO recruits seasoned advisers", "content": "A black punching bag hovers alongside the desk in Grant Verstandig's office. A pair of worn sneakers rests on the arm of a white, modular sofa. Overhead, a photo of a hulking Muhammad Ali hangs from the wall.\n\nIf it weren't for the panoramic view of the Georgetown waterfront, this space could be mistaken for a college dorm room. Perhaps that's fitting for the 22-year-old chief executive of Audax Health, a start-up that blends social media with health care. The company has been Verstandig's brainchild since the District native endured a spate of intensive knee surgeries to correct sports-induced injuries. Its banner product, called Careverge, will make a public debut next week at the Consumer Electronics Show in Las Vegas.\n\nCareverge users answer a series of questions about their health history that range from daily dietary habits to specific chronic illnesses. The site allows them to anonymously read relevant Web resources, connect with similar users and set health goals. Audax plans to market Careverge as a benefit for companies to offer employees.Heavy-hitting backers\n\nWhile Washington has become home to a crop of 20-something entrepreneurs with ambitious plans to launch businesses, Audax may stand out for the seasoned lineup of mentors Verstandig has managed to recruit to its board.\n\nJohn Sculley, the former chief executive of Pepsi and Apple, has been a financial backer and business adviser since May 2011. He had been hunting for a health care investment when a business contact introduced him to Audax.\n\nFrom the health arena, Verstandig has brought on Dr. Richard Klausner, former executive director for global health of the Bill and Melinda Gates Foundation and director of the National Cancer Institute from 1995 to 2001.\n\nKlausner, who worked with Verstandig's mother in the Clinton administration, introduced him to the health and science fields as a high school student through summer work at the National Institutes of Health.\n\nAlso on the board are Roger W. Ferguson Jr.,president and chief executive of retirement services provider TIAA-CREF, and John Wallis Rowe, the former chairman and chief executive of insurance firm Aetna.Knee injury sparked idea\n\nVerstandig said the makings of the business really began in his Brown University dorm room where, while laid up after a knee operation, he began to compile a list of industry contacts. Then through social media, e-mail and telephone calls, he began to network.\n\n\"The candid truth is spending a lot of time not being able to move made you focus,\" said Verstandig, who would later drop out to focus on the company full time.\n\nHis persistence and charm - it's clear Verstandig can talk his way through almost any social situation - impressed Sculley. During their meeting, Sculley twice asked Verstandig to name his biggest mistake. His response - setting unrealistic expectations and misreading progress - aren't uncommon for first-time entrepreneurs.\n\n\"In every case, except this one, I always work with serial entrepreneurs,\" Sculley said. \"I said, 'Gee, this violates everything I said I'm going to do. I'm not going to work with people who have never built companies before and yet here is a guy who resembles in some ways Steve Jobs and Bill Gates when they were in their 20s.'\"\n\nFor a maturity beyond his years, Verstandig certainly looks his age. Smelling of cologne and wearing a V-neck sweater and dark denim jeans, his office attire could transition easily to a bar in Foggy Bottom.\n\nBut casual is to be expected at a company like Audax. Ping-pong tables, remote-control helicopters and oversized bean-bag chairs are just a few of the start-up staples that the company makes available to its 55 employees.No more do-it-all himself\n\nVerstandig admits that his position has come with a steep learning curve, particularly as a first-time CEO without any prior business experience or education.\n\n\"Back in the early days I did everything myself because I thought I could do it faster, better, quicker, but now I just hire people who are smarter and hire people who are more experienced,\" he said.\n\nAnd then there are challenges beyond his control. Health care can be a notoriously stubborn market where attempts at innovation become bogged down by bureaucracy, regulation and big business.\n\n\"You have to be acutely aware of all those things that are swirling around, but at the end of the day, one thing I've learned from my mentors is you can't control what you can't control,\" Verstandig said.\n\nTiming, however, may be on the company's side. Verstandig and Sculley both believe that health care reform at the federal level combined with other initiatives to revamp the system make now an opportune moment for a company like Audax to make a strong play.\n\n\"I am absolutely convinced that the health care problem that we have in the economy will eventually be solved largely by innovation from the private sector and not the government,\" Sculley said.\n\noverlys#washpost.com", "publishedDate": "2012-01-02T00:00:00Z", "harvestDate": "2012-01-02T00:00:00Z", "estimatedPublishedDate": "2012-01-02T00:00:00Z", "url": "https://some.url.com/?a=24414529721&p=6e2&v=1&x=F98g2RdEJP-tyqi1V-39-A", "originalUrl": "http://some.url.com/noarticleurl?type=Companydnf&lnlni=54MC-P5M1-JBFW-C23W-00000-00", "wordCount": "822", "copyright": "Copyright 2012 The Washington Post. All Rights Reserved.", "duplicateGroupId": "24414529721", "media": {}, "publishingPlatform": {"itemId": "54MC-P5M1-JBFW-C23W-00000-00"}, "adultLanguage": "false", "topics": [{"name": "Executive moves news"}], "indexTerms": [{"domains": ["IND"], "name": "SOCIAL MEDIA", "score": "75"}, {"domains": ["IND"], "name": "CONSUMER ELECTRONICS", "score": "74"}, {"domains": ["SUB"], "name": "EXECUTIVES", "score": "90"}, {"domains": ["SUB"], "name": "STUDENT HOUSING", "score": "89"}, {"domains": ["SUB"], "name": "KNEE DISORDERS & INJURIES", "score": "89"}, {"domains": ["SUB"], "name": "WOUNDS & INJURIES", "score": "78"}, {"domains": ["SUB"], "name": "RESEARCH INSTITUTES", "score": "78"}, {"domains": ["SUB"], "name": "MEDICAL RESEARCH", "score": "76"}, {"domains": ["SUB"], "name": "HEALTH DEPARTMENTS", "score": "76"}, {"domains": ["SUB"], "name": "ENTREPRENEURSHIP", "score": "75"}, {"domains": ["SUB"], "name": "CHRONIC DISEASES", "score": "75"}, {"domains": ["SUB"], "name": "SPORTS INJURIES", "score": "74"}, {"domains": ["SUB"], "name": "FOUNDATIONS", "score": "73"}, {"domains": ["SUB"], "name": "BOARD CHANGES", "score": "70"}, {"domains": ["SUB"], "name": "CANCER", "score": "69"}, {"domains": ["SUB"], "name": "NUTRITION", "score": "68"}, {"domains": ["SUB"], "name": "TALKS & MEETINGS", "score": "65"}, {"domains": ["SUB"], "name": "TRADE SHOWS", "score": "54"}], "companies": [{"name": "Global Health Ltd", "symbol": "WSY", "exchange": "BER", "isin": "AU000000GLH2", "titleCount": 0, "contentCount": 1, "primary": true}, {"name": "Global Health Ltd", "symbol": "GLH", "exchange": "ASX", "isin": "AU000000GLH2", "titleCount": 0, "contentCount": 1, "primary": true}, {"name": "Space Co Ltd", "symbol": "9622", "exchange": "TKS", "isin": "JP3400050005", "titleCount": 0, "contentCount": 1, "primary": true}], "semantics": {"entities": [{"properties": [{"name": "type", "value": "Company"}, {"name": "value", "value": "Aetna.Knee"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Company"}, {"name": "value", "value": "Audax Group"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Company"}, {"name": "value", "value": "PepsiCo"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Company"}, {"name": "value", "value": "Audax Health"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Company"}, {"name": "value", "value": "Apple Inc."}], "provider": "3"}, {"properties": [{"name": "type", "value": "Company"}, {"name": "value", "value": "TIAA"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Steve Jobs"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "John Rowe"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Richard Klausner"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Grant Verstandig"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "John Sculley"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Muhammad Ali"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Roger W. Ferguson, Jr."}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Bill Gates"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Hillary Clinton"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Person"}, {"name": "value", "value": "Careverge"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Organization"}, {"name": "value", "value": "Brown University"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Organization"}, {"name": "value", "value": "National Institutes of Health"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Organization"}, {"name": "value", "value": "Bill & Melinda Gates Foundation"}], "provider": "3"}, {"properties": [{"name": "type", "value": "Organization"}, {"name": "value", "value": "National Cancer Institute"}], "provider": "3"}]}, "sentiment": {"score": "0.08718867"}, "print": {"publicationEdition": "Every Edition", "pageNumber": "A06"}, "author": {"publishingPlatform": {}}, "licenses": [{"name": "Company Licensed"}], "linkedArticles": [], "source": {"id": "93252", "name": "The Washington Post", "homeUrl": "http://www.washingtonpost.com/", "category": "National", "editorialRank": "1", "location": {"country": "United States", "countryCode": "US", "region": "Americas", "subregion": "Northern America", "state": "District of Columbia"}, "metrics": {"mozscape": {}}, "feed": {"id": "8528", "mediaType": "Print", "publishingPlatform": "Company Licensed", "idFromPublisher": "783", "language": "Unassigned", "rank": {"inboundLinkCount": "0"}, "inWhiteList": "false"}}}]}
Thanks

Answer 1

This should work. Let me know if you have any questions
spark = SparkSession.builder.getOrCreate()
df = spark.read.json("test.json")
df.createOrReplaceTempView("test")
spark.sql("select article.adultLanguage, company.* from test lateral view explode(articles) as article lateral view explode(article.companies) as company ").show(10,False)