I modified the https://developers.google.com/drive/v2/reference/files/list example to get a list of all file items, using request partial resources.
def retrieve_all_files(service, fields = None):
"""Retrieve a list of File resources.
Args:
service: Drive API service instance.
fields: The fields (of each files Resource item!), e.g. 'id,title'
if not passed, get everything.
Returns:
List of File resources.
"""
result = []
page_token = None
while True:
try:
param = {}
if fields:
param['fields'] = 'nextPageToken,items(' + fields + ')'
if page_token:
param['pageToken'] = page_token
files = service.files().list(**param).execute()
result.extend(files['items'])
page_token = files.get('nextPageToken')
if not page_token:
break
except errors.HttpError, error:
print 'An error occurred: %s' % error
break
return result
In my test Drive folder I have a file that has a description string set. When I run
retrieve_all_files(drive_service)
without setting the fields parameter (so get everything), sure enough in the list of items, the file looks fine:
[{u'alternateLink': u'...',...
u'description': u'Description',
u'id': u'...',...},...]
but running
retrieve_all_files(drive_service, fields='id,description,title')
yields a list where none of the items have the 'description' property. Not all files have the 'description' property set; is this why the partial resource response is omitting the field even from items that have the property? I'm not sure if this is an undocumented feature that I didn't expect or a bug.
I was able to reproduce your issue. When using an authorized scopes list of:
https://www.googleapis.com/auth/drive
which should be full access, I was not able to get the description field returned. However, I found that if I also included the scope:
https://www.googleapis.com/auth/drive.appdata
the description field would return properly.
Can you confirm on your end that adding the above to your scopes resolves the issue? Seems like a bug on Google's end.
This was confirmed as a bug and a fix should be available in a few days.
Related
On our Stack users by default have discoverer permissions on resources. I was surprised that this also gives users the ability to query the last_transaction_rid of the dataset (using catalog/datasets/<rid>/reverse-transactions2/<branch>), so this method to check if a user has access is not working.
What would be the recommended and most performant API call to check if I, with my current Foundry token, can read the actual content of a dataset? Note: I don't want to query the content, but just understand if I would have the permissions to do so.
Would Data Lineage App (Monocle) work for you?
Open it workspace/data-integration/monocle/ > Find the dataset(s) > top right dropdown ("Resource Type" -> "Permissions")
The API that powers it can be extracted by checking the Dev Tools > Network Tab if you want / need to automate it (I've done this in the past and worked well for my use case)
I have resorted back to calling the compass/resources API and checking if response['operations'] contains compass:view:
def get_dataset_details(api_base: str, dataset_path_or_rid: str, headers: dict) -> dict:
"""
Returns the resource information of a dataset
Args:
dataset_path_or_rid: The full path or rid to the dataset
Returns: (dict) the json response of the api
"""
if 'ri.foundry.main.dataset' in dataset_path_or_rid:
response = requests.get(f"{api_base}/compass/api/resources/{dataset_path_or_rid}",
headers=headers,
params={'decoration': 'path'})
else:
response = requests.get(f"{api_base}/compass/api/resources",
headers=headers,
params={
'path': dataset_path_or_rid,
'decoration': 'path'
})
response.raise_for_status()
if response.status_code != 200:
raise ValueError(f"Dataset {dataset_path_or_rid} not found; "
f"If you are sure your dataset_path is correct, "
f"check if your jwt token "
f"is still valid!")
return response.json()
details = get_dataset_details(...)
if 'compass:view' not in dataset_details['operations']:
raise ValueError("No compass:view access to dataset")
I'm trying to create a view to import a csv using drf and django-import-export.
My example (I'm doing baby steps and debugging to learn):
class ImportMyExampleView(APIView):
parser_classes = (FileUploadParser, )
def post(self, request, filename, format=None):
person_resource = PersonResource()
dataset = Dataset()
new_persons = request.data['file']
imported_data = dataset.load(new_persons.read())
return Response("Ok - Babysteps")
But I get this error (using postman):
Tablib has no format 'None' or it is not registered.
Changing to imported_data = Dataset().load(new_persons.read().decode(), format='csv', headers=False) I get this new error:
InvalidDimensions at /v1/myupload/test_import.csv
No exception message supplied
Does anyone have any tips or can indicate a reference? I'm following this site, but I'm having to "translate" to drf.
Starting with baby steps is a great idea. I would suggest get a standalone script working first so that you can check the file can be read and imported.
If you can set breakpoints and step into the django-import-export source, this will save you a lot of time in understanding what's going on.
A sample test function (based on the example app):
def test_import():
with open('./books-sample.csv', 'r') as fh:
dataset = Dataset().load(fh)
book_resource = BookResource()
result = book_resource.import_data(dataset, raise_errors=True)
print(result.totals)
You can adapt this so that you import your own data. Once this works OK then you can integrate it with your post() function.
I recommend getting the example app running because it will demonstrate how imports work.
InvalidDimensions means that the dataset you're trying to load doesn't match the format expected by Dataset. Try removing the headers=False arg or explicitly declare the headers (headers=['h1', 'h2', 'h3'] - swap in the correct names for your headers).
I am dealing with creating AWS API Gateway. I am trying to create CloudWatch Log group and name it API-Gateway-Execution-Logs_${restApiId}/${stageName}. I have no problem in Rest API creation.
My issue is in converting restApi.id which is of type pulumi.Outout to string.
I have tried these 2 versions which are proposed in their PR#2496
const restApiId = apiGatewayToSqsQueueRestApi.id.apply((v) => `${v}`);
const restApiId = pulumi.interpolate `${apiGatewayToSqsQueueRestApi.id}`
here is the code where it is used
const cloudWatchLogGroup = new aws.cloudwatch.LogGroup(
`API-Gateway-Execution-Logs_${restApiId}/${stageName}`,
{},
);
stageName is just a string.
I have also tried to apply again like
const restApiIdStrign = restApiId.apply((v) => v);
I always got this error from pulumi up
aws:cloudwatch:LogGroup API-Gateway-Execution-Logs_Calling [toString] on an [Output<T>] is not supported.
Please help me convert Output to string
#Cameron answered the naming question, I want to answer your question in the title.
It's not possible to convert an Output<string> to string, or any Output<T> to T.
Output<T> is a container for a future value T which may not be resolved even after the program execution is over. Maybe, your restApiId is generated by AWS at deployment time, so if you run your program in preview, there's no value for restApiId.
Output<T> is like a Promise<T> which will be eventually resolved, potentially after some resources are created in the cloud.
Therefore, the only operations with Output<T> are:
Convert it to another Output<U> with apply(f), where f: T -> U
Assign it to an Input<T> to pass it to another resource constructor
Export it from the stack
Any value manipulation has to happen within an apply call.
So long as the Output is resolvable while the Pulumi script is still running, you can use an approach like the below:
import {Output} from "#pulumi/pulumi";
import * as fs from "fs";
// create a GCP registry
const registry = new gcp.container.Registry("my-registry");
const registryUrl = registry.id.apply(_=>gcp.container.getRegistryRepository().then(reg=>reg.repositoryUrl));
// create a GCP storage bucket
const bucket = new gcp.storage.Bucket("my-bucket");
const bucketURL = bucket.url;
function GetValue<T>(output: Output<T>) {
return new Promise<T>((resolve, reject)=>{
output.apply(value=>{
resolve(value);
});
});
}
(async()=>{
fs.writeFileSync("./PulumiOutput_Public.json", JSON.stringify({
registryURL: await GetValue(registryUrl),
bucketURL: await GetValue(bucketURL),
}, null, "\t"));
})();
To clarify, this approach only works when you're doing an actual deployment (ie. pulumi up), not merely a preview. (as explained here)
That's good enough for my use-case though, as I just want a way to store the registry-url and such after each deployment, for other scripts in my project to know where to find the latest version.
Short Answer
You can specify the physical name of your LogGroup by specifying the name input and you can construct this from the API Gateway id output using pulumi.interpolate. You must use a static string as the first argument to your resource. I would recommend using the same name you're providing to your API Gateway resource as the name for your Log Group. Here's an example:
const apiGatewayToSqsQueueRestApi = new aws.apigateway.RestApi("API-Gateway-Execution");
const cloudWatchLogGroup = new aws.cloudwatch.LogGroup(
"API-Gateway-Execution", // this is the logical name and must be a static string
{
name: pulumi.interpolate`API-Gateway-Execution-Logs_${apiGatewayToSqsQueueRestApi.id}/${stageName}` // this the physical name and can be constructed from other resource outputs
},
);
Longer Answer
The first argument to every resource type in Pulumi is the logical name and is used for Pulumi to track the resource internally from one deployment to the next. By default, Pulumi auto-names the physical resources from this logical name. You can override this behavior by specifying your own physical name, typically via a name input to the resource. More information on resource names and auto-naming is here.
The specific issue here is that logical names cannot be constructed from other resource outputs. They must be static strings. Resource inputs (such as name) can be constructed from other resource outputs.
Encountered a similar issue recently. Adding this for anyone that comes looking.
For pulumi python, some policies requires the input to be stringified json. Say you're writing an sqs queue and a dlq for it, you may initially write something like this:
import pulumi_aws
dlq = aws.sqs.Queue()
queue = pulumi_aws.sqs.Queue(
redrive_policy=json.dumps({
"deadLetterTargetArn": dlq.arn,
"maxReceiveCount": "3"
})
)
The issue we see here is that the json lib errors out stating type Output cannot be parsed. When you print() dlq.arn, you'd see a memory address for it like <pulumi.output.Output object at 0x10e074b80>
In order to work around this, we have to leverage the Outputs lib and write a callback function
import pulumi_aws
def render_redrive_policy(arn):
return json.dumps({
"deadLetterTargetArn": arn,
"maxReceiveCount": "3"
})
dlq = pulumi_aws.sqs.Queue()
queue = pulumi_aws.sqs.Queue(
redrive_policy=Output.all(arn=dlq.arn).apply(
lambda args: render_redrive_policy(args["arn"])
)
)
The following action:
def addMembers(){
Map result = [message:"successful"]
try {
def group = Group.get(params.id)
def json = request.JSON
def users = json.users.collect{Usr.get(it.id)}
result.members = groupService.addMembers(group,users)
}catch(Exception e){
message = "Exception $e"
result.message = message
response.setStatus(hsr.SC_METHOD_NOT_ALLOWED)
}
respond result, [model:[result:result]]
}//eo addMember
In conjunction with the following addMembers.gson file
model {
Map result
}
json{
message result.message
members g.render(template:"simpleMember", collection: result.members, var:'member')
}
Gets a null pointer exception:
java.lang.NullPointerException: Cannot get property 'message' on null object
I have things working well in other actions where I respond domain objects but the client side needs a message if I catch an exception and it really didn't seem worth it to create an arbitrary pogo when [message:"",members:[]] could do the same amount of work as an arbitrary extra file and an extra 5-10 lines of code.
Update 1
I tried replacing Map with an arbitrary ResultHolder class to placate any strict typing that might have been in play.
That didn't help
Update 2
In my .gson file I replaced
json{
with
json g.render(result){
And that gets me null output instead of null pointers.
Equally unacceptable.
Update 3
In order to try to assess how to interact with gson template without depending on ajax posts and database interaction I make the following arbitrary action:
def jsonDbug(){
def result = [message:"hi"]
respond result, [model:[jsonDbug:result]]
}
and an arbitrary gson file:
model{
Map jsonDebug
}
json{
says jsonDebug.message
}
This allows me to make changes faster to see what is going wrong.
I'm trying calling in other ways except respond now but nothing works.
It's as if JSON views are strictly for domain objects and nothing else.
It turns out that I wasn't that very far off when I started.
The problem was:
respond result, [model:[result:result]]
Which is the predominant example used at http://views.grails.org/latest/
When I changed to:
render(view:"addMembers", model:[result:result])
It worked exactly how I wanted it to.
I would like calls to /contacts/1.json to return json, 1.api to return browsableAPI, and calls with format=None aka /contacts/1/ to return a template where we call render_form. This way end-users can have pretty forms, and developers can use the .api format, and ajax/apps etc use .json. Seems like a common use case but something isn't clicking for me here in DRF...
Struggling with how DRF determines the Renderer used when no format is given. I found and then lost some info here on stack exchange that basically said to split the responses based on format. Adding the TemplateHTMLRenderer caused all sorts of pain. I had tried to split based on format but that is giving me JSON error below.
I don't understand the de facto way to define what renderer should be used. Especially when no format is provided. I mean, it "just works" when using Response(data). And I can get the TemplateHTMLRenderer to work but at the cost of having no default Renderer.
GET /contacts/1/ Gives the error:
<Contact: Contact object> is not JSON serializable
Using this code:
class ContactDetail(APIView):
permission_classes = (permissions.IsAuthenticatedOrReadOnly,
IsOwnerOrReadOnly,)
queryset = Contact.objects.all()
renderer_classes = (BrowsableAPIRenderer, JSONRenderer, TemplateHTMLRenderer,)
"""
Retrieve, update or delete a contact instance.
"""
def get_object(self, pk):
try:
return Contact.objects.get(pk=pk)
except Contact.DoesNotExist:
raise Http404
def get(self, request, pk, format=None):
contact = self.get_object(pk)
serializer = ContactSerializer(contact)
if format == 'json' or format == "api":
return Response(serializer.data)
else:
return Response({'contact': contact, 'serializer':serializer}, template_name="contact/contact_detail.html")
But GET /contacts/1.json , 1.api, or 1.html ALL give me the correct output. So it seems that I have created an issue with the content negotiation for the default i.e. format=None
I must be missing something fundamental. I have gone through the 2 tutorials and read the Renderers docs but I am unclear on what I messed up here as far as the default. I am NOT using the DEFAULT_RENDERERS in settings.py, didn't seem to make a difference if in default or inside the actual class as shown above.
Also if anyone knows a way to use TemplateHTMLRenderer without needing to switch on format value, I'm all ears.
EDIT: IF I use
if format == 'json' or format == "api" or format == None:
return Response(serializer.data)
else:
return Response({'contact': contact, 'serializer':serializer},
Then I am shown the browsable API by default. Unfortunately, what I want is the Template HTML view by default, which is set to show forms for end users. I would like to keep the .api format for developers.
TL; DR: Check the order of your renderers - they are tried in order of declaration until a content negotiation match or an error occurs.
Changing the line
renderer_classes = (BrowsableAPIRenderer, JSONRenderer, TemplateHTMLRenderer, )
to
renderer_classes = (TemplateHTMLRenderer, BrowsableAPIRenderer, JSONRenderer, )
Worked for me. I believe the reason is because the content negotiator starts at the first element in the renderer classes tuple when trying to find a renderer. When I have format==None, I'm thinking there is nothing else for DRF to go on, so it assumes I mean the "default" which seems to be the first in the tuple.
EDIT: So, as pointed out by #Ross in his answer, there is also a global setting in the settings.py for the project. If I remove my class level renderer_classes declaration and instead use this in settings.py
# ERROR
REST_FRAMEWORK = {
'DEFAULT_RENDERER_CLASSES': (
'rest_framework.renderers.BrowsableAPIRenderer',
'rest_framework.renderers.JSONRenderer',
'rest_framework.renderers.TemplateHTMLRenderer',
)
}
Then I get a (different) JSON error. However, as long as
'rest_framework.renderers.BrowsableAPIRenderer',
is not listed first, for example:
# SUCCESS, even though JSON renderer is checked first
'DEFAULT_RENDERER_CLASSES': (
'rest_framework.renderers.JSONRenderer',
'rest_framework.renderers.TemplateHTMLRenderer',
'rest_framework.renderers.BrowsableAPIRenderer',
)
So if we hit BrowsableAPIRenderer before we try TemplateHTMLRenderer then we get an error - whether or not we are relying on renderer_classes or DEFAULT_RENDERER_CLASSES. I imagine it passes through JSONRenderer gracefully but for whatever reason BrowsableAPIRenderer raises an exception.
So I have simplified my view code after analyzing this...
def get(self, request, pk, format=None):
contact = self.get_object(pk)
serializer = ContactSerializer(contact)
if format == None:
return Response({'contact': contact, 'serializer':serializer}, template_name="contact/contact_detail.html")
else:
return Response(serializer.data)
..which better reflects what I was originally trying to do anyway.
When I look at the source code, the priority seems to be the order of the renderers specified in the DEFAULT_RENDERER_CLASSES parameter in settings.py:
REST_FRAMEWORK = {
'DEFAULT_RENDERER_CLASSES': (
'rest_framework.renderers.JSONRenderer',
'rest_framework.renderers.TemplateHTMLRenderer',
),
'DEFAULT_PARSER_CLASSES': (
'rest_framework.parsers.JSONParser',
'rest_framework.parsers.TemplateHTMLRenderer',
)
}
So, if you specify a bunch of renderer classes, the first renderer that is valid will be selected based on if it is valid for the request given the .json/.api/.html extension and the Accept: header (not content-type, as I said in the comment on your question).