i have this ugly noob func:
def run_prog(compile_container, user_input, container_file_path):
# bash doesnt support single quotes or quotes inside double quotes so ## = '
time_check_command = '/usr/bin/time -f "## , ##memory##:##%M## , ##time##:##%e##"'
result = compile_container.exec_run(
f"/bin/bash -c 'echo {user_input} | {time_check_command} ./a.out'",
workdir=container_file_path)[1].decode('utf-8')
result = '{"result":"' + result + ',}'
result = result.replace('##', '"')
result_dict = json.loads(result)
if result_dict['time'] == '0.00':
result_dict['time'] = '<0.01'
return result_dict
Most of time it retrun json like this:
{
"result": "888",
"memory": "1792",
"time": "<0.01"
}
But one time of 10 or mb 20 it raise an error "error json.decoder.JSONDecodeError: Expecting ','" and i dont know why. Same input always. Can u please tell what is wrong?
So, i dont really sure whats wrong, but switch from json.loads to ast.literal_eval helped.
result_dict = ast.literal_eval(result)
Related
I have been trying to load csv data into Snowflake using COPY INTO command
This is the sample data
4513194677~"DELL - ULTRASHARP 32\" MONITOR 4K U3223QE"~""~""
I have tried using below COPY INTO syntax
file_format =
type = 'csv'
field_delimiter = '~'
skip_header = 1
record_delimiter = '\\n'
field_optionally_enclosed_by = '"'
ESCAPE = 'NONE'
ESCAPE_UNENCLOSED_FIELD = 'NONE'
However, getting this error "Found character 'M' instead of field delimiter '~'"
How can I escape the " and load the columns data as DELL - ULTRASHARP 32 " MONITOR 4K U3223QE
If I try to use ESCAPE, I get below error when running the COPY command
[ERROR] ProgrammingError: 001003 (42000): 01a8e01d-3201-36a9-0050-4502537cfc7f: SQL compilation error:
syntax error line 15 at position 43 unexpected '''.
syntax error line 20 at position 20 unexpected ')'.
file_format =
type = 'csv'
field_delimiter = '~'
skip_header = 1
record_delimiter = '\\n'
field_optionally_enclosed_by = '"'
ESCAPE = '\\'
ESCAPE_UNENCLOSED_FIELD = '\\'
Try using two double quotes in the data instead of one without trying to escape the double quote
Data similar to "sample"
You can have your csv formated like below
"Data similar to ""sample"""
I'm trying to use an xcom_pull inside an SQL phrase executed by a Snowflake operator in Airflow.
I need the task_id name to use a variable since I want to support different tasks.
I tried this syntax but seems it is not being rendered ok.
Anyone has an idea how to do it?
This is the Python code:
for product, val in PRODUCTS_TO_EXTRACT_INC.items():
product_indicator, prefix = val
params['product_prefix'] = prefix
calculate_to_date = SnowflakeOperator(
dag=dag,
task_id=f'calculate_to_date_{prefix}',
snowflake_conn_id = SF_CONNECTION_ID,
warehouse=SF_WAREHOUSE,
database=BI_DB,
schema=STG_SCHEMA,
role=SF_ROLE,
sql= [ """
{SQL_FILE}
""".format(SQL_FILE="{% include '" + QUERIES_DIR + ETL + "/calculate_to_date.sql'" + " %}")
],
params=params
)
This is the SQL code for calculate_to_date.sql:
select '{{{{ (ti.xcom_pull(key="return_value", task_ids=["calculate_from_date_{}"])[0][0]).get("FROM_DATE") }}}}'.format(params.product_prefix) AS TO_DATE
This is the error message:
File "/home/airflow/gcs/dags/Test/queries/fact_subscriptions_events/calculate_to_date.sql", line 11, in template
select '{{{{ (ti.xcom_pull(key="return_value", task_ids=["calculate_from_date_{}"])[0][0]).get("FROM_DATE") }}}}'.format(params.product_prefix)
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'
the correct syntax is
select '{{ (ti.xcom_pull(key="return_value", task_ids="calculate_from_date_{}".format(params.product_prefix))[0]).get("FROM_DATE") }}' AS TO_DATE
it works like a charm
Using regex I extract a substring from a Sentence and then translate it multiple times but sometimes I getting an error.
My code:
from googletrans import Translator
try:
translator = Translator()
# substring Sentence: Direccion Referencia </tr>
# I want To Translate This Substring Sentence
# Address = re.search(r'(?<=Direccion).*?(?=</tr>)' ,
#new_get_htmlSource).group(0)
# Address Substring From Website
Address = re.search(r'(?<=<td>).*?(?=</td>)' , Address).group(0)
translated = translator.translate(str(Address))
Address = translated.text
except Exception as e :
exc_type , exc_obj , exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print("Error ON : ",sys._getframe().f_code.co_name + "--> " + str(e) ,
"\n" , exc_type , "\n" ,fname , "\n" , exc_tb.tb_lineno)
I expect the output of This: Reference # Translated Text
I get this error:
scraping_data--> Expecting value: line 1 column 1 (char 0)
class: json.decoder.JSONDecodeError
Scraping_Things.py
29
I am trying to write a method which returns a new value. Following code is modified from here:
| stripChars |
stripChars := [ :string :chars |
str := string reject: [ :c | chars includes: c ].
str displayNl. "THIS WORKS."
^ str "THIS DOES NOT WORK."
].
newstr := stripChars
value: 'She was a soul stripper. She took my heart!'
value: 'aei'.
newstr displayNl.
Although above function creates new string and displays it, there is error in returning or receiving returned new string:
$ gst make_fn_ques.st
Sh ws soul strppr. Sh took my hrt!
Object: 'Sh ws soul strppr. Sh took my hrt!' error: return from a dead method context
SystemExceptions.BadReturn(Exception)>>signal (ExcHandling.st:254)
SystemExceptions.BadReturn class(Exception class)>>signal (ExcHandling.st:151)
String(Object)>>badReturnError (Object.st:1389)
UndefinedObject>>executeStatements (make_fn_ques.st:10)
nil
Where is the problem and how can this be solved? Thanks for your help.
The
^ str
does not return from the block (stripChars), but from the enclosing method instead (non-local return).
Apparently GNU Smalltalk does not allow you to return from the script that you pass to gst in this way.
Just drop the ^, and keep only str as the last expression of the block. That will cause str to be the return value of the block.
I'm trying to run a query on Cassandra through spark.
When running this command:
val test = sc.cassandraTable[Person](keyspace,table)
.where("name=?","Jane").collect
I get the appropriate output for the query.
When I try to use the where statement to enter the query as a whole string I get an error.
I receive the query as a json:
{"clause": " name = 'Jane' "}
then turn it into a string.
When running
val query = (json \ "clause").get.as[String]
//turns json value into a string
val test = sc.cassandraTable[Person](keyspace,table)
.where(query).collect
I get the following error:
java.io.IOException: Exception during preparation of SELECT "uuid", "person", "age" FROM "test"."users" WHERE token("uuid") > ? AND token("uuid") <= ? AND name = Jane ALLOW FILTERING: line 1:232 no viable alternative at input 'ALLOW' (...<= ? AND name = [Jane] ALLOW...)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.createStatement(CassandraTableScanRDD.scala:288)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD.com$datastax$spark$connector$rdd$CassandraTableScanRDD$$fetchTokenRange(CassandraTableScanRDD.scala:302)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$18.apply(CassandraTableScanRDD.scala:328)
at com.datastax.spark.connector.rdd.CassandraTableScanRDD$$anonfun$18.apply(CassandraTableScanRDD.scala:328)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at com.datastax.spark.connector.util.CountingIterator.hasNext(CountingIterator.scala:12)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at com.datastax.spark.connector.util.CountingIterator.foreach(CountingIterator.scala:4)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
I suspect that when I turn the json value " name = 'Jane' " into a string, I lose the single quotes hence I get " name = Jane " which of course raises an error. I tried escaping the single quotes with \ and with a second pair of single quotes around the name Jane {"clause": " name = ''Jane'' "}. It doesn't solve the issue.
Edit: After further testing it's definitely the json that loses the single quotes and CQL needs them to perform the query. Can anyone suggest a way to escape/save the presence of the single quotes? I tried escaping with \ double single quotes '' . Is there a way to use JSON to provide proper whole CQL statements?
Please use Unicode character \u0027.