Efficiently load large JSON file into SQL Server - json

Question

I have been attempting to identify what the most effective way to load a large JSON file into SQL Server is.
I have a rather primitive API that most helpfully returns me a 150+ MB JSON string with ~450k rows and 12 columns. Ultimately I want to be able to use this in power-query, I figure the simplest way to store, index, then query it efficiently will be in SQL Server.
I have asked regarding filtering the data from the source (the most logical solution, but for the purposes of this question, count it out).
I have attempted code like the following to prove it will work
DECLARE #JSON VARCHAR(MAX)
select #JSON = BulkColumn
from OPENROWSET
(BULK 'C:\temp\test.txt', SINGLE_CLOB)
AS j
Select * FROM OPENJSON (#JSON)
The issue I have is that after 7 minutes query time on a laptop with fast SSD, 16gb ram and i7-7800hq I decided there must be a better way of going about this.
I'm happy to try any other language (say python, r, C# clr)
Security isn't my primary concern, I think I should be able to refresh the data in a few seconds rather than waiting several minutes.

Answer 1

I never used to encounter problems loading significant volumes (several hundreds of megabytes of json) in, simply a query variable from the C# app I wrote to load these files.. The load also included a query that only inserted records that didn't already exist.
INSERT dba.dbo.Table(columns,from,json)
FROM
OPENJSON (#json, '$.result' )
WITH (
column varchar(256) '$.a',
from varchar(256) '$.b',
json varchar(256) '$.c'
) j
LEFT JOIN dba.dbo.Table b
ON
j.column = b.column
WHERE
b.column IS NULL
Something like this would take around 30 seconds to load 100mb of json and insert only the new records. The c# looked like:
using (SqlCommand sc = new SqlCommand("...", new SqlConnection(_connstr)))
{
sc.Parameters.Add("#json", SqlDbType.NVarChar, -1).Value = json;
var prior = sc.Connection.State;
if (sc.Connection.State != ConnectionState.Open)
sc.Connection.Open();
int ins = sc.ExecuteNonQuery();
if (prior == ConnectionState.Closed)
sc.Connection.Close();
return ins;
}
Critically, I seldom needed to do what you're doing (retrieve all the rows back as a SSMS result set or whatever). I would do the load, then do the work I needed with them. An appreciable proportion of the time you're waiting might be down to SSMS retrieving millions of rows