I want to add a custom global macro to Airflow, I found this answer that outlines how this can be done. But I'm wondering if I can get the context implicitly (just like the built-in macros) e.g:
def ds_ny(**context):
"""
Custom macro that returns ds in NY timezone
"""
return context.get("execution_date").in_timezone("America/New_York").format("%Y-%m-%d")
Usage:
select * from table where some_date = '{{ macros.ds_ny }}'
Currently I have to pass a parameter to the macro for it to render {{ macros.ds_ny(execution_date) }}
Related
Version Information
vyper Version (output of vyper --version): 0.2.8+commit.069936f
OS: osx
Python Version (output of python --version): Python 2.7.16
Environment (output of pip freeze):
altgraph==0.10.2
bdist-mpkg==0.5.0
bonjour-py==0.3
macholib==1.5.1
matplotlib==1.3.1
modulegraph==0.10.4
numpy==1.8.0rc1
py2app==0.7.3
pyobjc-core==2.5.1
pyobjc-framework-Accounts==2.5.1
pyobjc-framework-AddressBook==2.5.1
pyobjc-framework-AppleScriptKit==2.5.1
pyobjc-framework-AppleScriptObjC==2.5.1
pyobjc-framework-Automator==2.5.1
pyobjc-framework-CFNetwork==2.5.1
pyobjc-framework-Cocoa==2.5.1
pyobjc-framework-Collaboration==2.5.1
pyobjc-framework-CoreData==2.5.1
pyobjc-framework-CoreLocation==2.5.1
pyobjc-framework-CoreText==2.5.1
pyobjc-framework-DictionaryServices==2.5.1
pyobjc-framework-EventKit==2.5.1
pyobjc-framework-ExceptionHandling==2.5.1
pyobjc-framework-FSEvents==2.5.1
pyobjc-framework-InputMethodKit==2.5.1
pyobjc-framework-InstallerPlugins==2.5.1
pyobjc-framework-InstantMessage==2.5.1
pyobjc-framework-LatentSemanticMapping==2.5.1
pyobjc-framework-LaunchServices==2.5.1
pyobjc-framework-Message==2.5.1
pyobjc-framework-OpenDirectory==2.5.1
pyobjc-framework-PreferencePanes==2.5.1
pyobjc-framework-PubSub==2.5.1
pyobjc-framework-QTKit==2.5.1
pyobjc-framework-Quartz==2.5.1
pyobjc-framework-ScreenSaver==2.5.1
pyobjc-framework-ScriptingBridge==2.5.1
pyobjc-framework-SearchKit==2.5.1
pyobjc-framework-ServiceManagement==2.5.1
pyobjc-framework-Social==2.5.1
pyobjc-framework-SyncServices==2.5.1
pyobjc-framework-SystemConfiguration==2.5.1
pyobjc-framework-WebKit==2.5.1
pyOpenSSL==0.13.1
pyparsing==2.0.1
python-dateutil==1.5
pytz==2013.7
scipy==0.13.0b1
six==1.4.1
xattr==0.6.4
this call of a for loop:
for i in range(self.some_uint256):
# do something...
is throwing the error:
StateAccessViolation: Value must be a literal
full error output:
vyper.exceptions.StateAccessViolation: Value must be a literalvyper.exceptions.StateAccessViolation: Value must be a literal
contract "vyper-farm/contracts/Farm.vy", function "_employ", line 152:4
151
---> 152 for i in range(self.num_employees):
-------------^
153 pass
what exactly am i doing wrong?
is this a misunderstanding as to what a literal is on my part?
Look at the description of range-function, there just one way to pass a variable to it:
for i in range(a, a + N):
pass
In your case it should be like this (not sure that it be useful):
num_employees: public(uint256)
#external
def __init__():
self.num_employees = 16
#external
def do_smth():
for i in range(self.num_employees, self.num_employees + 10):
pass
the issue above is not one of misunderstanding the for loop's use, instead it is a result of incompatible coding style with the security measures of vyper
the reason the for loop was being created was to make sure when an 'employee' was 'fired' or 'quit' then there wouldn't be an empty record in the list of 'employee's which was being maintained
instead, in order to avoid using a for loop altogether, and with a small sacrifice of not being able to remove no longer 'active employee's, best practice is to just track the 'employee' in a hashmap:
idToEmployee: HashMap[uint256, employee]
and when tracking the employees, simply assign an id attribute to the 'employee' using a global variable called num_employees
ex:
def employ():
new_employee: employee = employee ({
id: self.num_employees
})
self.idToEmployee[self.num_employees] = new_employee
when attempting to view or update that employee's info simply use:
self.idToEmployee[id]
#vladimir has a good explanation of how range is passed variables if there is still confusion about for loops in vyper for the reader
In fact, you can't use variables in range() directly, but we can use other method.
Here is my advice:
for i in range(999999):
if i < self.some_uint256:
# do something...
else:
break
I'm trying to format a jinja template parameter as an integer so I can pass it to an operator which expects INT (could be custom or PythonOperator) and I'm not able to.
See sample DAG below. I'm using the built-in Jinja filter | int but that's not working - the type remains <class 'str'>
I'm still new with Airflow but I don't think this is possible based on what I've read about Jinja/Airflow works. I see two main workarounds:
Change the operator parameter to expect string and handle the conversion underneath.
Handle this conversion in a separate PythonOperator which converts the string to an int and export that using xcom/task context. (I think this will work but not sure)
Please let me know of any other workarounds
def greet(mystr):
print (mystr)
print(type(mystr))
default_args = {
'owner': 'airflow',
'start_date': days_ago(2)
}
dag = DAG(
'template_dag',
default_args=default_args,
description='template',
schedule_interval='0 13 * * *'
)
with dag:
# foo = "{{ var.value.my_custom_var | int }}" # from variable
foo = "{{ execution_date.int_timestamp | int }}" # built in macro
# could be MyCustomOperator
opr_greet = PythonOperator(task_id='greet',
python_callable=greet,
op_kwargs={'mystr': foo}
)
opr_greet
Airflow 1.10.11
Updated answer:
As of Airflow 2.1, you can pass render_template_as_native_obj=True to the dag and Airflow will return the Python type (dict, int, etc) instead of string. See this pull request
dag = DAG(
dag_id="example_template_as_python_object",
schedule_interval=None,
start_date=days_ago(2),
render_template_as_native_obj=True,
)
Old answer for prior versions:
I found a related question that provides the best workaround, IMO.
Airflow xcom pull only returns string
The trick is to use a PythonOperator, do the datatype conversion there and then call the main operator with the parameter. Below is an example in converting a json string to a dict object. Same can apply for converting string to int, etc.
def my_func(ds, **kwargs):
ti = kwargs['ti']
body = ti.xcom_pull(task_ids='privious_task_id')
import_body= json.loads(body)
op = CloudSqlInstanceImportOperator(
project_id=GCP_PROJECT_ID,
body= import_body,
instance=INSTANCE_NAME,
gcp_conn_id='postgres_default',
task_id='sql_import_task',
validate_body=True,
)
op.execute()
p = PythonOperator(task_id='python_task', python_callable=my_func)
I believe Jinja is going always to return you a string: the template is a string and replacing values inside the template will return you a string.
If you are sure that foo is always an integer, you can do
opr_greet = PythonOperator(task_id='greet',
python_callable=greet,
op_kwargs={'mystr': int(foo)}
)
Update: it looks like Airflow uses the render method from Jinja2, which returns a Unicode string.
At this point, if you can modify greet, it is easier to manage the input parameter in that function.
I'm passing a context variable, x, into a template from a Djano view. It is a list of strings
x = ['Braselton', 'Buford']
Then I am using an ajax function to pass that variable back to a django view. The problem is when I retrieve that variable in a python view with the following code:
new_x = request.GET['x']
print(new_x)
I see the following:
['Braselton', 'Buford']
I've tried json.loads(request.GET['x']) and I keep getting the following error
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
Any help is much appreciated
You need to unescape those characters, there are lots of ways to do it..
Python Documentation for more info
import html.parser
import json
variable = "['Braselton', 'Buford']"
parser = html.parser.HTMLParser()
new_variable = parser.unescape(variable)
new_variable = json.loads(parser.unescape(new_variable).replace("'",'"')) # replace single quote
>>> ['Braselton', 'Buford'] # Type List
The problem is that python escaping the HTML elements. Note that it's not JSON.
to unescape you have to use HTML module.
import html
y = html.unenscape(new_x)
print(y) # output is ['Braselton', 'Buford']
Mark the variable as safe.
'{{ x | safe }}'
I'm new to Groovy and coding in general, but I've come a long way in a very short amount of time. I'm currently working in Confluence to create a tracking tool, which connects to a MySql Database. We've had some great success with this, but have hit a wall with using Groovy and the Run Macro.
Currently, we can use Groovy to populate fields within the Run Macro, which really works well for drop down options, example:
{groovy:output=wiki}
import com.atlassian.renderer.v2.RenderMode
def renderMode = RenderMode.suppress(RenderMode.F_FIRST_PARA)
def getSql = "select * from table where x = y"
def getMacro = '{sql-query:datasource=testdb|table=false} ${getSql} {sql-query}"
def get = subRenderer.render(getMacro, context, renderMode)
def runMacro = """
{run:id=test|autorun=false|replace=name::Name, type::Type:select::${get}|keepRequestParameters = true}
{sql:datasource=testdb|table=false|p1=\$name|p2=\$type}
insert into table1 (name, type) values (?, ?)
{sql}
{run}
"""
out.println runMacro
{groovy}
We've also been able to use Groovy within the Run Macro, example:
enter code here
{run:id=test|autorun=false|replace=name::Name, type::Type:select::${get}|keepRequestParameters = true}
{groovy}
def checkSql = "{select * from table where name = '\name' and type = '\$type'}"
def checkMacro = "{sql-query:datasource=testdb|table=false} ${checkSql} {sql-query}"
def check = subRenderer.render(checkMacro, context, renderMode)
if (check == "")
{
println("This information does not exist.")
} else {
println(checkMacro)
}
{groovy}
{run}
However, we can't seem to get both scenarios to work together, Groovy inside of a Run Macro inside of Groovy.
We need to be able to get the variables out of the Run Macro form so that we can perform other functions, like checking the DB for duplicates before inserting data.
My first thought is to bypass the Run Macro and create a simple from in groovy, but I haven't been too lucky with finding good examples. Can anyone help steer me in the right direction for creating a simple form in Groovy that would replace the Run Macro? Or have suggestions on how to get the rendered variables out of the Run Macro?
I am coding a custom Liquid tag as Jekyll plugin for which I need to preserve some values until the next invocation of the tag within the current run of the jekyll build command.
Is there some global location/namespace that I could use to store and retrieve values (preferably key-value pairs / a hash)?
You could add a module with class variables for storing the persistent values, then include the module in your tag class. You would need the proper accessors depending on the type of the variables and the assignments you might want to make. Here's a trivial example implementing a simple counter that keeps track of the number of times the tag was called in DataToKeep::my_val:
module DataToKeep
##my_val = 0
def my_val
##my_val
end
def my_val= val
##my_val = val
end
end
module Jekyll
class TagWithKeptData < Liquid::Tag
include DataToKeep
def render(context)
self.my_val = self.my_val + 1
return "<p>Times called: #{self.my_val}</p>"
end
end
end
Liquid::Template.register_tag('counter', Jekyll::TagWithKeptData)