carrierwave-aws: Rails API app file upload fails with NoMethodError - undefined method `match' for nil:NilClass - aws-sdk

Hitting quite the wall on this issue. Let me explain what i've done and show the snippets of configured code, to hopefully find a solution
Ruby: 2.4.0
Rails: 5.x
Gems (specific to this):
- carrierwave 1.2.3
- carrierwave-aws
Here is the error
NoMethodError - undefined method `match' for nil:NilClass:
app/controllers/api/v3/video_controller.rb:23:in `upload'
Here is my config/environments/carrierwave.rb file
CarrierWave.configure do |config|
config.storage :aws
config.aws_credentials = {
provider: "AWS",
aws_access_key_id: Rails.application.secrets.aws_access_id,
aws_secret_access_key: Rails.application.secrets.aws_access_secret,
region: "us-east-1"
}
config.aws_bucket = "{{BUCKET_NAME_HERE}}"
end
Here is my uploader
class VideoUploader < CarrierWave::Uploader::Base
storage :aws
end
Finally, here is how i'm calling the upload, since it's an API, i'm not using a form builder or model assignment.
def upload
video = params[:video]
if video
# Send file to Amazon S3 bucket and grab URL
uploader = VideoUploader.new
uploader.store!(video.tempfile)
# uploader.retrieve_from_store!(tempfile_name)
# logger.info uploader.download_url(video.original_filename)
end
end
I've tried carrierwave + fog, fog-aws. When I remove the S3 equation the file will be stored locally (development machine). I am at my wits end with trying lib/ hacks, downgrading versions. I hope its a simple oversight but am out of ideas.
Any help is greatly appreciated!

Fixed! The problem was pretty stupid and man I wish I caught it sooner. The initializer file was in the environments directory (I even listed it in my question!), not the initializers directory... don't code on an empty stomach.
Would be great if the code could detect that missing file... but at least its solved.

Related

ChainableTemporaryCredentials getPromise and Missing credentials in config, if using AWS_CONFIG_FILE

I have an node application deployed in GCP.
The application includes code to access ressources in AWS-cloud.
For this purpose it uses the aws-SDK with ChainableTemporaryCredentials.
The relevant code lines are...
const credentials = new ChainableTemporaryCredentials({
params: {
RoleArn: `arn:aws:iam::${this.accountId}:role/${this.targetRoleName}`,
RoleSessionName: this.targetRoleName,
},
masterCredentials: new WebIdentityCredentials({
RoleArn: `arn:aws:iam::${this.proxyAccountId}:role/${this.proxyRoleName}`,
RoleSessionName: this.proxyRoleName,
WebIdentityToken: token,
}),
})
await credentials.getPromise()
The WebIdentityToken was received from google and looks good.
At AWS-side I created an proxy-role (the line from masterCredentials RoleArn).
However at runtime I get the error:
Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1
I do not understand this error. Because my application runs in GCP and I use temporary credentials I do not understand why I should use aws-credentials in form of an credentials file or environment variables like AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY. I thought the idea to use ChainableTemporaryCredentials is NOT to have direct aws-credentials. Right?
You can see the public code at:
https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/aws/src/application/GCPCredentials.ts
and documentation regarding env-variables at:
https://www.cloudcarbonfootprint.org/docs/configurations-glossary/
Any help which leads to understanding of this error message is welcome.
Thomas
Solved it. "Missing credentials in config, if using AWS_CONFIG_FILE, set AWS_SDK_LOAD_CONFIG=1 was totally misleading." In reality it was a problem with the field-names in the GCP-JWT-token und the policy in aws. See https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_iam-condition-keys.html#ck_aud

flask_restplus: unable to get flask_swagger_ui to work in conjunction with namespaces

We have a large flask_restplus application that's been in service for a year or so. We now would like to use flask_swagger_ui to work within this application to serve some statically generated swagger docs for part of the application.
Our flask_restplus-based application is already using namespaces, but the flask_swagger_ui docs that I have found seem to suggest that we have to use blueprints via its get_swaggerui_blueprint method in order to serve this swagger. However, when we do this, the flask_restplus application doesn't recognize the URL associated with this blueprint. We're wondering if perhaps this might be due to some sort of unwanted interaction between namespaces and blueprints within the same application ... or perhaps could it just be due to the fact that we're doing something incorrectly ... ???
Here is what I believe is the pertinent fragment of our code ...
app = flask.Flask(__name__)
api = flask_restplus.Api(
app,
version='1.0',
title='Test Application',
description='Test Application',
doc='/docs/'
)
ns = api.namespace('server', description='Main server', path='/server')
utilns = api.namespace('util', description='Util', path='/util')
SWAGGER_URL = '/swagger'
API_URL = 'http://petstore.swagger.io/v2/swagger.json' # just for testing
swaggerui_blueprint = get_swaggerui_blueprint(
SWAGGER_URL,
API_URL,
config={
'app_name': "Test application"
},
)
app.register_blueprint(swaggerui_blueprint)
# Set up endpoints under the namespaces defined above,
# and do some more initialization.
app.run()
However, if we try to access our app like this, http://our.domain.name:PORT/swagger/dist, it fails as follows ...
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 FINAL//EN">
<title>405 Method Not Available</title>
<h1>Method Not Allowed</h1>
<p>This method is not allowed for the requested URL.</p>
We get the same error if we leave off "dist", as follows:
http://our.domain.name:PORT/swagger.
This also fails if we refer to a static swagger.json file instead of the petshop URL.
All the other namespace-defined endpoints work fine.
Can anyone see what we are doing incorrectly?
Thank you very much in advance.
I stumbled upon the solution. In flask_restplus, any blueprint and #app.route definitions must appear before the flask_restplus.Api instantiation. I couldn't find this in any documentation, and I got it from a discussion that involved someone who had encountered and finally fixed this issue. So, this is indeed a weird and seemingly undocumented interaction between blueprints and namespaces in flask_restplus.
Also, I discovered that url_prefix=SWAGGER_URL needs to be passed to app.register_blueprint along with the blueprint variable, itself.
Once I started doing the following, my swagger queries began to work:
app = flask.Flask(__name__)
# Any #app.route and blueprint definitions must
# appear before the flask_restplus.Api
# instantiation.
SWAGGER_URL = '/swagger'
# Use the following API_URL just for testing ...
API_URL = 'http://petstore.swagger.io/v2/swagger.json'
swaggerui_blueprint = get_swaggerui_blueprint(
SWAGGER_URL,
API_URL,
config={
'app_name': "Test application"
},
)
app.register_blueprint(
swaggerui_blueprint,
url_prefix=SWAGGER_URL
)
# OK. Now we can instantiate flask_restplus.API
api = flask_restplus.Api(
app,
version='1.0',
title='Test Application',
description='Test Application',
doc='/docs/'
)
ns = api.namespace(
'server',
description='Main server',
path='/server'
)
utilns = api.namespace(
'util',
description='Util',
path='/util'
)
# Set up endpoints under the namespaces defined above,
# and do some more initialization. And then ...
app.run()

JRuby error loading private key with jruby-openssl in bouncycastle

I'm trying to sign emails sent by my application with an s/mime certificate. The application uses JRuby 1.7.26 and the newest version of the jruby-openssl gem. (Version 0.9.19).
When I try to load my private key like this:
key_file = File.read(Rails.root.join('config', 'domain.key'))
private_key = OpenSSL::PKey::RSA.new(key_file)
I get this error:
Java::JavaLang::NoSuchMethodError: org.bouncycastle.util.encoders.Base64.encode([BII)[B
from org.jruby.ext.openssl.x509store.PEMInputOutput.writeEncoded(PEMInputOutput.java:1496)
...
If that works i would like to sign the emails like this:
data = "Test"
OpenSSL::PKCS7.write_smime(OpenSSL::PKCS7.sign(certificate, private_key, data, [], OpenSSL::PKCS7::DETACHED))
I googled a lot to find solutions for this error or how to sign emails with JRuby in general but I could not find anything helpful.
Thank you in advance for any help.
likely means a mess up in class-path -> BC .jars loaded from somewhere else.
JRuby-OpenSSL's packed BC (in 0.9.19 it is 1.55) has the needed Base64 method - otherwise would fail to compile the Ruby extension bits in the first place.

Parsoid test page fail during VisualEditor installation

I'm trying to install VisualEditor in my MediaWiki wiki but I get stuck when I test Parsoid.
This is the result of the test page:
error: No API URI available for prefix: enwiki; domain: undefined path: /_rt/mediawikiwiki/Parsoid
Error: No API URI available for prefix: enwiki; domain: undefined
at /usr/lib/parsoid/src/lib/config/MWParserEnvironment.js:295:10
at /usr/lib/parsoid/node_modules/prfun/lib/index.js:532:26
at tryCatch2 (/usr/lib/parsoid/node_modules/babybird/lib/promise.js:48:12)
at PrFunPromise.Promise (/usr/lib/parsoid/node_modules/babybird/lib/promise.js:458:15)
at new PrFunPromise (/usr/lib/parsoid/node_modules/prfun/lib/index.js:57:21)
at /usr/lib/parsoid/node_modules/prfun/lib/index.js:530:18
at tryCatch1 (/usr/lib/parsoid/node_modules/babybird/lib/promise.js:40:12)
at promiseReactionJob (/usr/lib/parsoid/node_modules/babybird/lib/promise.js:269:19)
at PromiseReactionJobTask.call (/usr/lib/parsoid/node_modules/babybird/lib/promise.js:284:3)
at flush (/usr/lib/parsoid/node_modules/babybird/node_modules/asap/raw.js:50:29)
I set the API in the settings.js file end to make sure it is correct I tested using the curl command. And it works.
But I still have the problem.
Any suggestion?
You would've put something like this in Parsoid's localsettings.js:
parsoidConfig.setInterwiki( 'localhost', 'http://mediawiki.krenair.dev/mediawiki_dev/w/api.php' );
(example from my dev wiki setup)
That first string (in my case, 'localhost') should be identical to the value VE is set to use by $wgVisualEditorParsoidPrefix in your wiki's LocalSettings.php (unless you're using some other system to configure that stuff like VirtualRestConfig, in which case I can probably help in the comments). I believe you currently have it set to 'enwiki' for some reason, or else something is going wrong leading parsoid to default to 'enwiki' (I really don't know why they consider that a sane default).

Download a file from Google Drive using google-api-ruby-client

I try to download files from a directory on Google Disk.
The code, mostly copied from official quickstart guide, works fine:
# ... code from official quickstart tutorial...
# Initialize the API
service = Google::Apis::DriveV3::DriveService.new
service.client_options.application_name = APPLICATION_NAME
service.authorization = authorize
# now the real deal
response = service.list_files(q: "'0ByJUN4GMe_2jODVJVVpmRE1VTDg' in parents and trashed != true",
page_size: 100,
fields: 'nextPageToken, files(id, name)')
puts 'Files:'
puts 'No files found' if response.files.empty?
response.files.each do |file|
puts "#{file.name} (#{file.id})"
# content = service.get_file(file.id, download_dest: StringIO.new)
end
The output looks fine:
Files:
k.h264 (0B4D93ILRdf51Sk40UzBoYmZKMTQ)
output_file.h264 (0B4D93ILRdf51V1RGUDFIWFQ5cG8)
test.mp4 (0B4D93ILRdf51dWpoZWdBV3l4WjQ)
test2.mp4 (0B4D93ILRdf51ZmN4ZGlwZjBvR2M)
test3.mp4 (0B4D93ILRdf51ZXo0WnVfdVBjTlk)
12.mp4 (0ByJUN4GMe_2jRzEwS1FWTnVkX00)
01.mp4 (0ByJUN4GMe_2jSlRWVEw4a1gxa2s)
01.mp4 (0ByJUN4GMe_2jOFpPMW9YNjJuY2M)
But once I uncomment content = service.get_file(file.id, download_dest: StringIO.new), I get a lot of errors:
Files:
k.h264 (0B4D93ILRdf51Sk40UzBoYmZKMTQ)
/Users/mvasin/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-api-client-0.9.15/lib/google/apis/core/http_command.rb:211:in `check_status': Invalid request (Google::Apis::ClientError)
[...lots of other 'from' stuff...]
from /Users/mvasin/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/google-api-client-0.9.15/generated/google/apis/drive_v3/service.rb:772:in `get_file'
from quickstart.rb:56:in `block in <main>'
from quickstart.rb:54:in `each'
from quickstart.rb:54:in `<main>'
But that's the way it should work according to ruby section in "download files" official tutorial.
I also tried content = service.get_file(file.id, download_dest: "/tmp/#{file.name}"), but it failed with the same set of errors.
UPDATE: Here are my findings. If you start with Google Drive API Ruby quick start tutorial, and want make it download something,
1) change scope to not just let your script read meatadata, but read files contents as well:
SCOPE = Google::Apis::DriveV3::AUTH_DRIVE_METADATA_READONLY
to at least
SCOPE = Google::Apis::DriveV3::AUTH_DRIVE_READONLY
2) Filter out google docs files, because you can't download them this way, you have to convert them. To filer them:
2.1) Add mime_type to fileds set:
response = service.list_files(page_size: 10, fields: 'nextPageToken, files(id, name, mime_type)')
2.2) and in the final loop where you print files' ids and names, put something like
service.get_file(file.id, download_dest: "/your/path/#{file.name}") unless file.mime_type.include? "application/vnd.google-apps"
From the error that you got, it says that your request is invalid. So make sure that your request there are correct. Here is the documentation on how to download files using Ruby(just click the Ruby on the example to view the ruby code.)
Take NOTE: Downloading the file requires the user to have at least
read access. Additionally, your app must be authorized with a
scope
that allows reading of file content.
For more information, check these threads:
How to download file from google drive api
A Ruby library to read/write files