LuaLaTex using fontspec package and luacode reading JSON file - json

I'm using Latex since years but I'm new to embedded luacode (with Lualatex). Below you can see a simplified example:
\begin{filecontents*}{data.json}
[
{"firstName":"Max", "lastName":"Möller"},
{"firstName":"Anna", "lastName":"Smith"}
];
\end{filecontents*}
\documentclass[11pt]{article}
\usepackage{fontspec}
%\setmainfont{Carlito}
\usepackage{tikz}
\usepackage{luacode}
\begin{document}
\begin{luacode}
require("lualibs.lua")
local file = io.open('data.json','rb')
local jsonstring = file:read('*a')
file.close()
local jsondata = utilities.json.tolua(jsonstring)
tex.print('\\begin{tabular}{cc}')
for key, value in pairs(jsondata) do
tex.print(value["firstName"] .. ' & ' .. value["lastName"] .. '\\\\')
end
tex.print('\\hline\\end{tabular}')
\end{luacode}
\end{document}
When executing Lualatex following error occurs:
LuaTeX error [\directlua]:6: attempt to index field 'json' (a nil value) [\directlua]:6: in main chunk. \end{luacode}
When commenting the line \usepackage{fontspec} the output will be produced. Alternatively, the error can be avoided by commenting utilities.json.tolua(jsonstring) and all following lua-code lines.
So the question is: How can I use both "fontspec" package and json-data without generating an error message? Apart from this I have another question: How to enable german umlauts in output of luacode (see first "lastName" in example: Möller)?
Ah, I'm using TeX Live 2015/Debian on Ubuntu 16.04.
Thank you,
Jerome

Related

Octave - error reading data with textscan function

I am trying to read data of the following format with textscan:
date,location,new_cases,new_deaths,total_cases,total_deaths
2019-12-31,Afghanistan,0,0,0,0
2020-01-01,Afghanistan,0,0,0,0
2020-01-02,Afghanistan,0,0,0,0
2020-01-03,Afghanistan,0,0,0,0
2020-01-04,Afghanistan,0,0,0,0
...
(Full data file available here: https://covid.ourworldindata.org/data/ecdc/full_data.csv)
My code is:
# Whitespace replaced with _
file_name = "full_data.csv";
fid = fopen(file_name, "rt");
data= textscan(fid, "%s%s%d%d%d%d", "Delimiter", ",", "HeaderLines", 1, ...
"ReturnOnError", 0);
fclose(fid);
Text scan terminates with an error:
error: textscan: Read error in field 3 of row 421
Row 421 is the center row in the example below:
2020-01-12,Australia,0,0,0,0
2020-01-13,Australia,0,0,0,0
2020-01-14,Australia,0,0,0,0
2020-01-15,Australia,0,0,0,0
2020-01-16,Australia,0,0,0,0
2020-01-17,Australia,0,0,0,0
2020-01-18,Australia,0,0,0,0
I've checked the row it complains about and there is nothing different from the example above. I've replaced all spaces in the file with underscores too. Am I doing something wrong with textcan?

Python 3 psycopg2 COPY from stdin failed: error in .read()

I am trying to apply the code found on this page, in particular part 'Copy Data from String Iterator' of the Table of Contents, but run into an issue with my code.
Since not all lines coming from the generator (here log_lines) can be imported into the PostgreSQL database, I try to filter the correct lines (here row) using itertools.filterfalse like in the codeblock below:
def copy_string_iterator(connection, log_lines) -> None:
with connection.cursor() as cursor:
create_staging_table(cursor)
log_string_iterator = StringIteratorIO((
'|'.join(map(clean_csv_value, (
row['date'],
row['time'],
row['cs_uri_query'],
row['s_contentpath'],
row['sc_status'],
row['s_computername'],
...
row['sc_substates'],
row['s_port'],
row['cs_version'],
row['c_protocol'],
row.update({'cs_cookie':'x'}),
row['timetakenms'],
row['cs_uri_stem'],
))) + '\n')
for row in filterfalse(lambda line: "#" in line.get('date'), log_lines)
)
cursor.copy_from(log_string_iterator, 'log_table', sep = '|')
When I run this, cursor.copy_from() gives me the following error:
QueryCanceled: COPY from stdin failed: error in .read() call
CONTEXT: COPY log_table, line 112910
I understand why this error happens, it is because in the test file I use there are only 112909 lines that meet the filterfalse condition. But why does it try to copy line 112910 and throw the error and not just stop?
Since Python doesn't have a coalescing operator, add something like:
(map(clean_csv_value, (
row['date'] if 'date' in row else None,
:
row['cs_uri_stem'] if 'cs_uri_stem' in row else None,
))) + '\n')
for each of your fields so you can handle any missing fields in the JSON file. Of course the fields should be nullable in the db if you use None otherwise replace with None with some default value for that field.

Python JSON Decoding With Asynchat Cannot Catch ValueError Exception

I can't seem to catch an exception when using json.loads even though I specifically call it out. I largely see this when trying to stress my server with lots of client connection sending data very quickly. See my error below:
(I've replaced my IP address with X's in the error code)
EX: Unterminated string starting at: line 1 column 49 (char 48) Data:
'{"ap-hdop":0.55,"rtcmin":"38","ap-latdec":3.134,"a' error: uncaptured
python exception, closing channel
(:Unterminated string
starting at: line 1 column 49 (char 48)
[//faraday_server_handler.py|collect_incoming_data|34]
[/usr/lib/python2.7/json/init.py|loads|338]
[/usr/lib/python2.7/json/decoder.py|decode|366]
[/usr/lib/python2.7/json/decoder.py|raw_decode|382])
I understand this that the code fails because I simply miss a double quotes on the line:
'{"ap-hdop":0.55,"rtcmin":"38","ap-latdec":3.134,"a'
This line is usually a LOT longer so that "a.... was supposed to keep going and complete it's quotes.
Here's my relevant code:
def collect_incoming_data(self, data):
"""Read an incoming message from the client, place JSON message data into buffer"""
#self.logger.debug('collect_fing_data() -> (%d bytes)\n"""%s"""', len(data), data)
try:
loaded_data = json.loads(data)
except ValueError, ex:
self.handle_error()
type,value,traceback = sys.exc_info()
print type
#print "Collect Incoming Data: " . time.strftime("%Y-%m-%d %H:%M:%S",time.gmtime())
print "EX: ", ex
print "Data: ",repr(data)
Any ideas? I scoured the internet to see if I can find this issue, but I appear to be setting up to capture the exception which everyone else having this issue with loads seems to suggest to do.
EDIT 3/1/2016 - Evening
Commenting out my exception handle_error() let me see more of the error:
except ValueError, ex:
self.handle_error()
type,value,traceback = sys.exc_info()
print type
#print "Collect Incoming Data: " . time.strftime("%Y-%m-%d %H:%M:%S",time.gmtime())
print "EX: ", ex
print "Data: ",repr(data)
Below is my new error, I've commented out personal data. It's apparent that the issue I really have now is in-fact the unterminated string
EX: Unterminated string starting at:
line 1 column 49 (char 48) Data:
'{"ap-hdop":0.55,"rtcmin":"31","ap-latdec":XX.XXX,"a' EX: No JSON object could be decoded Data:
'p-latdeg":34,"adc6":2006,"adc7":2007,"adc4":2004,"adc5":2005,"adc2":2002,"adc3":2003,"adc0":2000,"adc1":2001,"gpio-0":30,"gpio-1":50,"gpio-2":70,"speed":5.0,"adc8":2008,"rtcday":"01","longdeg":118,"longdec":XX.XXX,"altitude":31.0,"ap-speed":0.0,"ap-pdop":0.77,"lat-dir":"N","long-dir":"W","hdop":0.01,"ap-rf":0,"alt-units":"M","rtcdow":"2","callsign":"XXXXX","ap-callsign":"XXXXX","id":1,"ap-id":1,"rtcyear":"2016","rtcmon":"03","ap-vdop":0.66,"ap-lat-dir":"N","vdop":0.02,"rtchour":"22","latdec":XX.XXX,"latdeg":34,"ap-longdeg":118,"ap-longdec":XX.XXX,"rtcsec":"15","ap-altitude":86.0,"ap-long-dir":"W","pdop":0.01,"ap-alt-units":"M","faraday-port":0}'
OK my original question was answered which was "Why am I not catching the ValueError exception even though I am providing code to do just that?"
#tadhg McDonald-jensen was correct with his comment to remove my call to handle_error().
I still have some other issues but they are a different question. Thanks!

download a csv zip file

I'm trying to download the following file. The code works just find in RStudio when I run to the console. But when I try to compile a markdown file (to either html or pdf), it gives an error. Why can't markdown communicate with the csv zip file?
```
temp = tempfile()
download.file("http://www.cms.gov/Research-Statistics-Data-and-Systems/Statistics-Trends-and-Reports/Medicare-Provider-Charge-Data/Downloads/Inpatient_Data_2013_CSV.zip", temp)
temp2 = unz(temp, "Medicare_Provider_Charge_Inpatient_DRG100_FY2013.csv")
medData = read_csv(temp2)
```
Gives the following error (I had to remove URLs in error because I don't have enough reputation points):
trying URL Quitting from lines 41-49 (medicare.Rmd) Error in
download.file("..", : cannot open URL '...' Calls: ...
withCallingHandlers -> withVisible -> eval -> eval -> download.file

Error in fromJSON(paste(raw.data, collapse = "")) : unclosed string

I am using the R package rjson to download weather data from Wunderground.com. Often I leave the program to run and there are no problems, with the data being collected fine. However, often the program stops running and I get the following error message:
Error in fromJSON(paste(raw.data, collapse = "")) : unclosed string
In addition: Warning message:
In readLines(conn, n = -1L, ok = TRUE) :
incomplete final line found on 'http://api.wunderground.com/api/[my_API_code]/history_20121214pws:1/q/pws:IBIRMING7.json'
Does anyone know what this means, and how I can avoid it since it stops my program from collecting data as I would like?
Many thanks,
Ben
I can recreate your error message using the rjson package.
Here's an example that works.
rjson::fromJSON('{"x":"a string"}')
# $x
# [1] "a string"
If we omit a double quote from the value of x, then we get the error message.
rjson::fromJSON('{"x":"a string}')
# Error in rjson::fromJSON("{\"x\":\"a string}") : unclosed string
The RJSONIO package behaves slightly differently. Rather than throwing an error, it silently returns a NULL value.
RJSONIO::fromJSON('{"x":"a string}')
# $x
# NULL