I am trying to use templating in headers value of my http request using my custom http operator ( Extends simpleHttpOperator ). Seems like templating is supported only in data field. How can implemente the same in headers field. I wanted to pass authorization header templated. Please find my code below.
import airflow
from airflow import DAG
from airflow.configuration import conf
from airflow.operators.python_operator import PythonOperator
from airflow.operators.auth_plugins import SipAuthOperator
from airflow.operators.util_plugins import AuthUtility
DEFAULT_VERSION = '2.0'
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': airflow.utils.dates.days_ago(2),
'email': ['airflow#example.com'],
'email_on_failure': False,
'email_on_retry': False
}
DAG_ID = 'test_dag'
dag = DAG(DAG_ID, default_args=default_args,
schedule_interval=None,
catchup=False)
dag.doc_md = __doc__
auth_endpoint = conf['auth_apis']['authenticate_end_point']
def inspect_params(**context):
token = context['task_instance'].xcom_push(key='JWT_TOKEN',value='helloooo'
)
print(token)
test_operator = PythonOperator(dag=dag,task_id='init_api',
python_callable=inspect_params,
provide_context=True, )
# data={'token':'{{task_instance.xcom_pull(key=\'JWT_TOKEN\')}}'}
# {'Authorization':'Bearer '+'{{task_instance.xcom_pull(key=\'JWT_TOKEN\')}}'
http_operator = SipAuthOperator( dag=dag,task_id='authenticate_api',http_conn_id='auth_api',endpoint=auth_endpoint,headers={'token':'{{task_instance.xcom_pull(key=\'JWT_TOKEN\')}}'})
test_operator >> http_operator
Header value coming as {'token': "{{task_instance.xcom_pull(key='JWT_TOKEN')}}"} which is not as desred. If I put the same value in data field it works fine as expected. Is jinja templating supported on headers ? Any work around for this issue ?
The template_fields attribute in an operator determines which parameters can be templated. For example, in the original SimpleHttpOperator you can see the following
class SimpleHttpOperator(BaseOperator):
...
template_fields = ('endpoint', 'data',)
...
Which is why endpoint and data are supported template fields. Similarly in your custom operator, you'll want to include header.
class SipAuthOperator(SimpleHttpOperator):
...
template_fields = ('endpoint', 'data', 'header',)
...
Related
I know this question has been answered before, but I seem to have a different problem. Up until a few days ago, my querying of YouTube never had a problem. Now, however, every time I query data on any video the rows of actual video data come back as a single empty array.
Here is my code in full:
# -*- coding: utf-8 -*-
import os
import google.oauth2.credentials
import google_auth_oauthlib.flow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from google_auth_oauthlib.flow import InstalledAppFlow
import pandas as pd
import csv
SCOPES = ['https://www.googleapis.com/auth/yt-analytics.readonly']
API_SERVICE_NAME = 'youtubeAnalytics'
API_VERSION = 'v2'
CLIENT_SECRETS_FILE = 'CLIENT_SECRET_FILE.json'
def get_service():
flow = InstalledAppFlow.from_client_secrets_file(CLIENT_SECRETS_FILE, SCOPES)
credentials = flow.run_console()
#builds api-specific service
return build(API_SERVICE_NAME, API_VERSION, credentials = credentials)
def execute_api_request(client_library_function, **kwargs):
response = client_library_function(
**kwargs
).execute()
print(response)
columnHeaders = []
# create a CSV output for video list
csvFile = open('video_result.csv','w')
csvWriter = csv.writer(csvFile)
csvWriter.writerow(["views","comments","likes","estimatedMinutesWatched","averageViewDuration"])
if __name__ == '__main__':
# Disable OAuthlib's HTTPs verification when running locally.
# *DO NOT* leave this option enabled when running in production.
os.environ['OAUTHLIB_INSECURE_TRANSPORT'] = '1'
youtubeAnalytics = get_service()
execute_api_request(
youtubeAnalytics.reports().query,
ids='channel==UCU_N4jDOub9J8splDAPiMWA',
#needs to be of form YYYY-MM-DD
startDate='2018-01-01',
endDate='2018-05-01',
metrics='views,comments,likes,dislikes,estimatedMinutesWatched,averageViewDuration',
dimensions='day',
filters='video==ZeY6BKqIZGk,YKFWUX9w4eY,bDPdrWS-YUc'
)
You can see in the Reports: Query front page that you need to use the new scope:
https://www.googleapis.com/auth/youtube.readonly
instead of the old one:
https://www.googleapis.com/auth/yt-analytics.readonly
After changing the scope, perform a re-authentication (delete the old credentials) for the new scope to take effect.
This is also confirmed in this forum.
One of the mishaps may come if you chose wrong account/accounts during oAuth2 authorisation. For instance you may have to get "account" on the firs screen but then on second screen (during authorisation) use "brand account" and not the main account from the first step that also is on a list for second step.
I got the same problem and replacing with https://www.googleapis.com/auth/youtube.readonly scope doesn't work.
(Even making requests in the API webpage, it returns empty rows.)
Instead, using the https://www.googleapis.com/auth/youtube scope works fine in my case.
Basically, I am trying to turn raw SQL query result to JSON object and then send it to client-side via AJAX.
Here is my view (I am using Django 1.8.6)
import MySQLdb
from django.db import connection
import json
from django.http import HttpResponse
from django.core import serializers
def test_view(request):
cursor = connection.cursor()
cursor.execute("select id, name from okved")
data = cursor.fetchall
json_data = serializers.serialize('json', data)
return HttpResponse(json_data, content_type="application/json")
The respective URLConf
url(r'^test/$', test_view),
JQuery function
var test = function()
{
$.ajax({
type:"GET",
url:"/test",
dataType : 'json',
cache: "false",
data:{},
success:function(response)
{
alert("Test successful");
}
});
return true;
}
I am constantly getting GET http://127.0.0.1:8000/test/ 500 (INTERNAL SERVER ERROR) error, wheareas I follow all the recommendations that I have come across the previous threads here. I would really appreciate any help on this. I have blown my mind trying surfing Stackoverflow on this.
First of all, 500 error is just an error code, django would give you stacktrace of where the error happens in the function stack, you should learn to read that and find where the error happens.
From you code sounds like you are trying to use serializers to serialize a raw query result. This wouldn't work because the raw result is a tuple of tuples, each sub tuple is a record returned from your database. Django serializers is only good at serializing objects queried from ORM. You'd better go with json.dumps() method. Here's last few lines in your views.py:
data = cursor.fetchall()
json_data = json.dumps(data)
return HttpResponse(json_data, content_type="application/json")
Here's the doc for django serializer.
I am trying to do something very simple but haven't found how to do it yet.
I have a model and an endpoint returning a JSON array reprenseting the instances of this model with Django Rest Framework. I want to include the JSON in an HTML template (for SEO and for fast initial data loading). Something like
<script>
var data = {% json_from_django_rest_framework "mymodel" %};
</script>
Is there an easy way to do this? Should I just go a different way?
Another way of doing this, which gets around rendering the view.
In your views.py;
class FooDetailView(DetailView):
model = Foo
template_name = 'foo/detail.html'
def get_context_data(self, **kwargs):
bars = []
for bar in self.object.bars.all():
bars.append(BarSerializer(bar).data)
kwargs['bars'] = JSONRenderer().render(bars)
return super(FooDetailView, self).get_context_data(**kwargs)
And in your template;
<script>
var bars = {{ bars|safe }};
</script>
It should really go without saying that you should pay attention to potential performance issues with this approach, ie.. perhaps it's best to paginate bars.
As discussed in the comments, here is an example of how to reuse the result from your api endpoint in a normal Django view by using Django's resolve function.
views.py
import json
from django.core.urlresolvers import resolve
from django.views.generic.base import View
class FooView(View):
def get(self, request):
# optional stuff in your view...
##
# Resolving another Django view programmatically
##
rev = '/path/to/api/endpoint/' # or use reverse()
view, vargs, vkwargs = resolve(rev)
vkwargs['request'] = request
res = view(*vargs, **vkwargs)
c = Context({
'data': json.dumps(res.data)
})
# Now the JSON serialized result from the API endpoint
# will be available in the template variable data.
return render(request, 'my-app/my-template.html', c)
my-template.html
<script>
var data = {{ data }};
</script>
Note 1: Instead of hardcoding the path in rev = '/path/to/api/endpoint/' it is better to reverse() the url, but I left it out to remove that as a source for errors. If you are going this direction, here is a list of the default url names provided by DRF
Note 2: The snippet would benefit from exception handling, like making sure that res returned 200, has the data property, etc.
i want to have custom json response after data post sendind to my Tastypie API models django.
class MyModelResource(ModelResource):
my_field=""
class Meta:
queryset = MyModel.objects.all()
resource_name = 'nick_name'
authentication = ApiKeyAuthentication()
authorization = DjangoAuthorization()
def hydrate(self, bundle):
#on recupere les donnée injectée par bundle.data['title']
#et on inject les donnée via bundle.obj.title
#bundle.data['my_field'] ="1234"
bundle.obj.my_field=bundle.data['my_field']
self.my_field = bundle.data['my_field']
return bundle
def wrap_view(self, view):
"""
Wraps views to return custom error codes instead of generic 500's
"""
#csrf_exempt
def wrapper(request, *args, **kwargs):
try:
callback = getattr(self, view)
response = callback(request, *args, **kwargs)
if request.is_ajax():
patch_cache_control(response, no_cache=True)
lst_dic=[]
mon_dic = dict(success=True, my_field=self.my_field
)
# response is a HttpResponse object, so follow Django's instructions
# to change it to your needs before you return it.
# https://docs.djangoproject.com/en/dev/ref/request-response/
lst_dic.append(mon_dic)
response = HttpResponse(simplejson.dumps(lst_dic), content_type='application/json')
return response
except (BadRequest, fields.ApiFieldError), e:
return HttpBadRequest({'success':False,'code': 666, 'message':e.args[0]})
except ValidationError, e:
# Or do some JSON wrapping around the standard 500
return HttpBadRequest({'success':False,'code': 777, 'message':', '.join(e.messages)})
except Exception, e:
# Rather than re-raising, we're going to things similar to
# what Django does. The difference is returning a serialized
# error message.
return self._handle_500(request, e)
return wrapper
My problem here, i can't grab the self.my_field value to put in mon_dic, i always have data object, not value...
thx for help
EDIT : Add my_field global variable, and then grab value from bundle that's it ;)
Maybe I am not understanding what you want to do here. But wrap_view is for handling customer error responses. If all you want to do is return the data that was posted, you can set always_return_data to true in your Meta:
class Meta:
always_return_data = True
Or if you want to control what data gets sent back, you can use the dehydrate method:
def dehydrate(self, bundle):
bundle.data['custom_field'] = "Whatever you want"
return bundle
Does anyone know how can I successfully retrieve the object count of a model, in JSON format, and how I need to configure my routing? I'm trying to achieve this using a APIView and returning a Response formatted by JSONRenderer.
UPDATE:
#api_view(['GET'])
#renderer_classes((JSONRenderer, JSONPRenderer))
def InfluenciasCountView(request, format=None):
influencia_count = Influencia.objects.count()
content = {'influencia_count': influencia_count}
return Response(content)
Here's the route I'm using:
url(r'^influencias/count/$', views.InfluenciasCountView, name='influencias-count')
Check out this snippet of code (the second one). If this does not suit your need, please add some of your code (for better understanding).
UPDATE
For routing, DRF offers a default router for each view. This means that you can have the following configuration in your urls.py: (using the example from the previous link)
url(r'^users/count/$', views. UserCountView.as_view(), name='users-count')
Then, when you access the URL your_base_url/users/count/ you will see something like {'user_count': 10}.
UPDATE 2
The entire code should look like this:
class UserCountView(APIView):
"""
A view that returns the count of active users.
"""
renderer_classes = (JSONRenderer, )
def get(self, request, format=None):
user_count = User.objects.count()
content = {'user_count': user_count}
return Response(content)
I am using routers from REST Framework to build my URLs. I tried above code but doesn't get it working. One of the problems is I cannot get /count/ into the router endpoints.
I checked DRF document (3.8.2) and found that there is a (new?) #action decorator (I was using 3.7.7 and it doesn't have it). So, here is my full solutions:
Upgrade DRF to 3.8.2 (or above) in requirements.txt (or PipFile if you using that).
Add a new action count method to the ModelViewSet
Update get_permissions to include the newly added action count
Here is my views.py:
from rest_framework.decorators import action
from rest_framework.response import Response
class PostViewSet(viewsets.ModelViewSet):
"""
API endpoint that allows recommend to be viewed or edited.
"""
model = Post
queryset = Post.objects.filter(is_active=True)
serializer_class = serializers.PostSerializer
filter_backends = (filters.SearchFilter, DjangoFilterBackend,)
search_fields = ('title', 'body',)
filter_fields = ('status', 'type')
def get_permissions(self):
if self.action in ('list', 'retrieve', 'create', 'count'):
return (AllowAny()),
if self.action in ('update', 'partial_update'):
return (IsAdminUser()),
return (IsAdminUser()),
#action(detail=False)
def count(self, request):
queryset = self.filter_queryset(self.get_queryset())
count = queryset.count()
content = {'count': count}
return Response(content)
To query count of posts: /api/posts/count/?format=json
To query count of published: /api/posts/count/?format=json&status=published
One of the important thing here is to use the queryset from filter_queryset(...), rather than Post.objects.all().
UPDATE
Since count is common, I created a mixin for that.
from rest_framework.decorators import action
from rest_framework.response import Response
class CountModelMixin(object):
"""
Count a queryset.
"""
#action(detail=False)
def count(self, request, *args, **kwargs):
queryset = self.filter_queryset(self.get_queryset())
content = {'count': queryset.count()}
return Response(content)
To use it, just add CountModelMixin to your ModelViewSet (also support nested ModelViewSet).
class PostViewSet(viewsets.ModelViewSet, CountModelMixin):
If you use permissions, also add 'count' to the list of granted action.