Stream a NOT NULL selection of a table? - sqlalchemy

Question

I'm trying to select the primary key for all rows in a table based on if another column is NULL.
The following code does not do what I want, but this is what it would look like as a pure select(), but the table is so large that it nearly fills up memory before returning any results.
s = tweets.select().where(tweets.c.coordinates != None)
result = engine.execute(s)
for row in result:
print(row)
Because the table is so large, I found a streaming solution that works for the session.query() object:
def page_query(q):
r = True
offset = 0
while r:
r = False
for elem in q.limit(1000).offset(offset):
r = True
yield elem
offset += 1000
so I'm trying to structure the above select() as a query(), but when I do, it returns every row in the table, including ones with coordinates = 'null'
q= session.query(Tweet).filter(Tweet.coordinates.is_not(None))
for i in page_query(q):
print(f' {i}')
If I instead do
q= session.query(Tweet).filter(Tweet.coordinates.is_not('null'))
for i in page_query(q):
print(f' {i}')
I get an error:
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "'null'"
LINE 3: WHERE milan_tweets.coordinates IS NOT 'null'
^
(using != appears to give the same results as the built in .is_not())
So how can I make this selection?
EDIT: Code block at the top does NOT do what I expected originally, my mistake.
Rows are added to the database as python Nones, and looking in dbeaver shows the values as "null"

Answer 1

You have correctly diagnosed the problem.
Query returns e.g. a million rows,
and the psycopg2 driver drags all
of those result rows over the network,
buffering them locally, before returning
even a single row up to your app.
Why? Because the public API includes a
detail where your app could ask
"how many rows were in that result?",
and the driver must retrieve all in order
to learn that bit of trivia.
If you promise not to ask the "how many?"
question, you can stream results with this:
import sqlalchemy as sa
engine = sa.create_engine(uri).execution_options(stream_results=True)
Then rows will be up-delivered to your app
nearly
as soon as they become available,
rather than being buffered
a long time.
This yields a significantly smaller
memory footprint for your python process,
as the DB driver layer does not need
to malloc() storage sufficient to
store all million result rows.
https://docs.sqlalchemy.org/en/14/core/connections.html#streaming-with-a-fixed-buffer-via-yield-per
cf test_core_fetchmany_w_streaming