we've built a web app that automatically indexes and saves email attachments directly to Google Drive based on user-defined business rules. We're seeing failures when syncing some Google Drive accounts and can't figure out what the problem is.
This is the error:
List Files Error: Backend Error
/mnt/opt/openera/current/app/models/cloud_account/google_drive.rb:299:in 'list_files'
/mnt/opt/openera/current/app/models/cloud_account/google_drive.rb:220:in 'do_discover_files'
/mnt/opt/openera/current/app/models/cloud_account/base.rb:304:in 'discover_files'
/mnt/opt/openera/current/app/workers/worker/file_synchronizer.rb:10:in 'perform'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:45:in 'block (3 levels) in
process'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:109:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:109:in 'block in in
voke'
/mnt/opt/openera/current/lib/sidekiq_repeat_jobs/middleware/server/repeat_jobs.rb:36:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/timeout.rb:14:in 'cal
l'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/active_record.rb:6:in
'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/retry_jobs.rb:49:in '
call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/logging.rb:11:in 'blo
ck in call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/logging.rb:22:in 'with_context'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/logging.rb:7:in 'call
'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:114:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:114:in 'invoke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:44:in 'block (2 levels) in
process'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:80:in 'stats'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:43:in 'block in process'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in 'public_send'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in 'dispatch'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/future.rb:18:in 'block in initiali
ze'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in 'block in c
reate'
UPDATE: Here is the actual code from our senior dev.
dev here (ffoeg) adding actual code invoked:
def list_files
pageToken = nil
params = {'maxResults' => 65535, 'q' => 'trashed = false'}
files = []
drive = session.discovered_api("drive", "v2")
while true do
call_params = pageToken.blank? ? params : params.merge('pageToken' => pageToken)
result = session.execute(api_method: drive.files.list, parameters: call_params)
if result.status == 200
files += result.data.items
break if result.data['nextPageToken'].blank?
pageToken = result.data['nextPageToken']
else
raise "List Files Error: #{result.data['error']['message']}"
end
end
files
end
UPDATING AGAIN:
We updated our code to get the exact error - here it is:
List Files Error: {"errors"=>[{"domain"=>"global", "reason"=>"backendError",
"message"=>"Backend Error"}], "code"=>503, "message"=>"Backend Error"}
The problem is the value of maxResults. It is far too large and will cause the request to time out. Unfortunately, a "safe" value is very small, about ~200 results per page. You can go larger, but results will vary based on the user. Users with smaller corpuses will probably work fine with larger page sizes, but people with larger corpuses appear to be more sensitive. Well, at least based on some very quick/informal testing on a few of my accounts.
FWIW, I remember things working better with larger page sizes. Will check if anything changed that might constrain things more than expected.
Related
I'm building a Memory Scanner and with some error handling I noticed that ReadProcessMemory() is reading 90% of process' pages, but the ones that have mbi.Protect value == 1 or 260 it fails and returns ERROR 299 (Partial Copy) and the output of BytesRead is 0.
I run it as admin, set debug privileges and open process with VM_READ, but these exactly pages with mbi.Protect == 260 and 1 are unreadable. So, it's normal that it cant read all pages or am I doing something wrong ? Here is the code: (to be reproducable it also need this part of code that I import to main code and its where I setup all the ctypes background: https://pastebin.com/hMxLej5k, then you open python, import the code below and write "main(pid)" where pid is the pid of the process you want to read).
from ctypes import *
from ctypes import wintypes
import win32security
from setup_apis import *
def setDebugPriv():
token_handle = wintypes.HANDLE()
if not OpenProcessToken(
GetCurrentProcess(),
TOKEN_ADJUST_PRIVILEGES | TOKEN_QUERY,
byref(token_handle),
):
print("Error:",kernel32.GetLastError())
return False
luidvalue = win32security.LookupPrivilegeValue ( None, win32security.SE_DEBUG_NAME )
if not win32security.LookupPrivilegeValue(
None,
win32security.SE_DEBUG_NAME ,
):
print("Error",kernel32.GetLastError())
return False
se_debug_name_value = LUID(luidvalue) # Valor local do Privilégio de Debug
LAA = LUID_AND_ATTRIBUTES (
se_debug_name_value,
SE_PRIVILEGE_ENABLED
)
tkp = TOKEN_PRIVILEGES (
1, # DWORD PrivilegeCount
LAA, # LUID_AND_ATTRIBUTES
)
if not AdjustTokenPrivileges(
token_handle,
False,
byref(tkp),
sizeof(tkp),
None,
None,
):
print("Error:",GetLastError)
CloseHandle(token_handle)
return False
return True
#################################
def main(pid=None):
setDebugPriv()
process = OpenProcess (
PROCESS_VM_READ|PROCESS_QUERY_INFORMATION,
False,
pid,
)
system_info = SYSTEM_INFO()
GetSystemInfo ( byref(system_info) )
MaxAppAdress = system_info.lpMaximumApplicationAdress
VirtualQueryEx = VirtualQueryEx64
mbi = MEMORY_BASIC_INFORMATION64()
memset (
byref(mbi),
0,
sizeof(mbi),
)
Adress = 0
BytesRead = c_size_t (0)
while MaxAppAdress > Adress:
VirtualQueryEx(
process,
Adress,
byref(mbi),
sizeof(mbi),
)
if mbi.State == MEM_COMMIT:
try:
ContentsBuffer = create_string_buffer(mbi.RegionSize)
except:
pass
if not ReadProcessMemory (
process,
Adress,
ContentsBuffer,
mbi.RegionSize,
byref(BytesRead),
):
print("Cant Read, Error: %i, Protect State: %i" %(kernel32.GetLastError(), mbi.Protect) )
print("BytesRead:", BytesRead)
Adress += mbi.RegionSize
continue
Adress += mbi.RegionSize
'''
See Memory Protection Constants (260 = 0x104). No access and page guard regions cause exceptions. You can't access a no_access and you don't want to fire page_guard exceptions as they are meant to warn a process that a stack needs to grow and commit more pages. Don't attempt to read them.
Constant
Value
Description
PAGE_NOACCESS
0x01
Disables all access to the committed region of pages. An attempt to read from, write to, or execute the committed region results in an access violation.This flag is not supported by the CreateFileMapping function.
PAGE_READWRITE
0x04
Enables read-only or read/write access to the committed region of pages. If Data Execution Prevention is enabled, attempting to execute code in the committed region results in an access violation.
PAGE_GUARD
0x100
Pages in the region become guard pages. Any attempt to access a guard page causes the system to raise a STATUS_GUARD_PAGE_VIOLATION exception and turn off the guard page status. Guard pages thus act as a one-time access alarm. For more information, see Creating Guard Pages.When an access attempt leads the system to turn off guard page status, the underlying page protection takes over.If a guard page exception occurs during a system service, the service typically returns a failure status indicator.This value cannot be used with PAGE_NOACCESS.This flag is not supported by the CreateFileMapping function.
I'm pretty new to programming ruby and using json packages, I think I'm facing a typical noob-error, but I just cant find my mistake for two days now.. I've started like this:
require 'discogs-wrapper'
require 'json'
aw = Discogs::Wrapper.new("my_application", user_token: "my_user_token")
inv = aw.get_user_inventory('my_user_name', :per_page => 1, :page => 1)
p=JSON.parse(inv)
What I recieve is this:
C:/Ruby24-x64/lib/ruby/2.4.0/json/common.rb:156:in `initialize': no implicit conversion of Hashie::Mash into String (TypeError)
from C:/Ruby24-x64/lib/ruby/2.4.0/json/common.rb:156:in `new'
from C:/Ruby24-x64/lib/ruby/2.4.0/json/common.rb:156:in `parse'
from C:/Users/rtuz2th/.RubyMine2017.3/config/scratches/scratch.rb:6:in `<top (required)>'
from -e:1:in `load'
from -e:1:in `<main>'
I've already been searching your forum and I've learned that this error probably means that I do not have a correct JSON-file (although it comes straight from the discogs-api, I basically thought these guys know what they do). So what I tried next was
require 'discogs-wrapper'
require 'json'
aw = Discogs::Wrapper.new("my_application", user_token: "my_user_token")
inv = aw.get_user_inventory('my_user_name', :per_page => 1, :page => 1)
inv = inv.to_json
p=JSON.parse(inv)
So at least, my code runs without any errors now. But when I try to refer to any information in inv, I just get empty responses. inv (without the .to_json looks like this:
{"pagination"=>
{"per_page"=>1,
"items"=>13692,
"page"=>1,
"urls"=>
{"last"=>
"URL containing tokens",
"next"=>
"URL containing tokens"},
"pages"=>13692},
"listings"=>
[{"status"=>"Draft",
"original_price"=>
{"curr_abbr"=>"EUR",
"formatted"=>"\u20AC10.46",
"value"=>10.46,
"curr_id"=>3},
"weight"=>230.0,
"original_shipping_price"=>
{"curr_abbr"=>"EUR",
"formatted"=>"\u20AC5.90",
"value"=>5.9,
"curr_id"=>3},
"price"=>{"currency"=>"EUR", "value"=>10.46},
"allow_offers"=>true,
"uri"=>"https://www.discogs.com/sell/item/437965910",
"sleeve_condition"=>"Very Good Plus (VG+)",
"format_quantity"=>1,
"id"=>437965910,
"shipping_price"=>{"currency"=>"EUR", "value"=>5.9},
"posted"=>"2017-02-07T23:43:01-08:00",
"ships_from"=>"Germany",
"in_cart"=>false,
"comments"=>"",
"seller"=>
{"username"=>"zweischeiben.de",
"stats"=>{"rating"=>"100.0", "total"=>143, "stars"=>5.0},
"uid"=>3359767,
"url"=>"https://api.discogs.com/users/zweischeiben.de",
"html_url"=>"https://www.discogs.com/user/zweischeiben.de",
"shipping"=>
"(Lots of information about shipping, cut just cut it out)"
"payment"=>"Bank Transfer, PayPal",
"avatar_url"=>
"(cut out url)",
"resource_url"=>"https://api.discogs.com/users/zweischeiben.de",
"id"=>3359767},
"condition"=>"Near Mint (NM or M-)",
"release"=>
{"thumbnail"=>
"(cut out url)",
"description"=>"Steamhammer - Mountains (LP, Album, RE)",
"artist"=>"Steamhammer",
"format"=>"LP, Album, RE",
"resource_url"=>"https://api.discogs.com/releases/7303333",
"title"=>"Mountains",
"year"=>0,
"id"=>7303333,
"catalog_number"=>"201.006, 0201.006"},
"resource_url"=>"https://api.discogs.com/marketplace/listings/437965910",
"audio"=>false,
"external_id"=>"3, 16",
"location"=>"T52574"}]}
But if I now try to refer to eg the location:
require 'discogs-wrapper'
require 'json'
aw = Discogs::Wrapper.new("my_application", user_token: "my_token")
inv=aw.get_user_inventory('my_username', :per_page => 1, :page => 1)
inv=inv.to_json
p=JSON.parse(inv)
puts p[location]
I'm just getting an error like this
C:\Ruby24-x64\bin\ruby.exe -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) C:/Users/rtuz2th/.RubyMine2017.3/config/scratches/scratch.rb
C:/Users/rtuz2th/.RubyMine2017.3/config/scratches/scratch.rb:9:in `<top (required)>': undefined local variable or method `location' for main:Object (NameError)
from -e:1:in `load'
from -e:1:in `<main>'
Process finished with exit code 1
or an empty response. I'm trying to solve this for two days now, but I'm absolutely out of ideas. Thanks for your help in advance!
inv = inv.to_json
p = JSON.parse(inv)
The above is a noop. Use inv as is, it’s a hash already.
puts p[location]
You cannot just lookup whatever key located deeply in the tree hierarchy or you hash. You hash has stings as keys, it has a two top-level elements, 'pagination' and 'listings', the latter is an array, and each element of this array is a hash having location key.
While you were cutting the long value, you dropped the ending comma, making me unable to copy-paste the input and provide the exact answer.
I moved the database to mysql which is on VM. After moving database I started getting this error when I am trying to access or saving data to database but some models updation are working fine.
The model I am accessing has some arithmetic computations which I feel is the reason for this issue.
Error is
undefined method `+#' for nil:NilClass
I tried from rails console as well and I am getting same error with this command
Settings.all
I guess this is part of that method method
if params[:lat] and params[:lon] and params[:rad] and !params[:lat].empty? and
!params[:lon].empty? and !params[:rad].empty?
lat = params[:lat].to_f
lon = params[:lon].to_f
rad = params[:rad].to_f
ear = 6371.00
min_lat = (lat) - (rad/ear)/180.0*Math::PI;
max_lat = (lat) + (rad/ear)/180.0*Math::PI;
min_lon = (lon) - (rad/ear/Math.cos(lon*Math::PI/180.0))/180.0*Math::PI;
max_lon = (lon) + (rad/ear/Math.cos(lon*Math::PI/180.0))/180.0*Math::PI;
#settings = #settings.includes(:settings_location)
.where("settings_locations.latitude > :min_lat AND " \
"settings_locations.latitude < :max_lat AND " \
"settings_locations.longitude > :min_lon AND " \
"settings_locations.longitude < :max_lon", {
min_lat: min_lat,
max_lat: max_lat,
min_lon: min_lon,
max_lon: max_lon});
stack trace
Started POST "/settings.json" for 127.0.0.1 at 2014-04-24 13:52:53 +0200
NoMethodError - undefined method `+#' for nil:NilClass:
app/models/settings.rb:54:in `<top (required)>'
activesupport (4.0.2) lib/active_support/dependencies.rb:424:in `block in load_file'
activesupport (4.0.2) lib/active_support/dependencies.rb:616:in `new_constants_in'
activesupport (4.0.2) lib/active_support/dependencies.rb:423:in `load_file'
activesupport (4.0.2) lib/active_support/dependencies.rb:324:in `require_or_load'
activesupport (4.0.2) lib/active_support/dependencies.rb:463:in `load_missing_constant'
setting.rb:
class Settings < ActiveRecord::Base
has_one :settings_location
def to_builder
Jbuilder.new do |json|
json.location settings_location.to_builder
end
end //this line is 54
Per the comments, there seem to be only a handful of cases where you can reproduce the same error message in IRB. The first is:
foo, bar = nil, nil
foo.send("+#", bar) # undefined method `+#' for nil:NilClass
The second is:
foo = nil
+foo # undefined method `+#' for nil:NilClass
In an eval context, the latter might more look like:
foo, bar = nil, nil
eval("foo = +#{bar.inspect}") # undefined method `+#' for nil:NilClass
My personal guess is that your problem is something along the lines of the latter. Not with eval, but rather with one of its plethora of Ruby variations such as define_method. The actual call is very likely somewhere in Rail's internals and related to some uninitialized variable — perhaps MySQL is returning null where SQLite was returning something due to a recent schema change that you didn't think about?
Personally, I'd try to narrow it down step by step:
1. Comment out:
json.location settings_location.to_builder
If the error is gone, then:
2.a) Replace it with:
builder = settings_location.to_builder
json.location builder
If the error message is still the same, then:
2.b) Try
Jbuilder.new
… without a block.
The point is, you want to narrow things down to the specific method that is causing the error, and then dive into that method's code to see what is nil and shouldn't be. (Perhaps the block is missing some kind of return value?)
Scenario: Testing with prompt popups
Given I am on the frames page
When I popup the prompt and enter "Jared Gatorboy"
SyntaxError: missing ; before statement (Selenium::WebDriver::Error::JavascriptError)
./features/support/pages/frames_page.rb:35:in `block in prompt_value'
./features/support/pages/frames_page.rb:34:in `prompt_value'
./features/step_definitions/web01_steps.rb:35:in `block (2 levels) in <top (required)>'
./features/step_definitions/web01_steps.rb:34:in `/^I popup the prompt and enter "(.*?)"$/'
features/web01.feature:20:in `When I popup the prompt and enter "Jared Gatorboy"'
Then the message from the prompt should read "enter your name"
And the default value from the prompt should be "John Doe"
./features/support/pages/frames_page.rb
def prompt_value(value)
in_frame(:id => 'frame_3') do |frame|
#prompt_response = prompt(value, frame) do
button_element(:id => 'prompt_button', :frame => frame).click
end
end
#prompt_response
end
./features/step_definitions/web01_steps.rb
When(/^I popup the prompt and enter "(.*?)"$/) do |value_to_enter|
on_page(FramesPage) do |page|
#prompt_response = page.prompt_value(value_to_enter)
end
end
App URL
Can you tell me what browser and driver you are using? In the page-object test suite there is a test that uses this page. Here is the step definition
When /^I trigger a prompt within a frame$/ do
#page.in_frame(:id => 'frame_3') do |frame|
#msg = #page.prompt("Cheezy", frame) do
#page.button_element(:id => 'prompt_button', :frame => frame).click
end
end
end
It looks to be the identical logic you have in your steps and the tests work with the latest firefox using both selenium-webdriver and watir-webdriver as the driver.
I am trying to take the results of a query to a db and save them to a csv file. Here is my current Ruby script.
#! /usr/bin/ruby
require 'rubygems'
require 'mysql'
require 'date'
# need mysql queries here / use mysql2 calls
db_con = Mysql.new("localhost", "root", "", "msd")
date_results = db_con.query("SELECT CONCAT(CONVERT(date_format(dd.date, '%b-%e'),char),'\\n') AS date
FROM msd.date_dim dd LEFT OUTER JOIN msd.results cs
ON dd.id = cs.date_id
GROUP BY
dd.date")
# create new xml file and insert header
open('test_trend.xml', 'w') do |f|
date_results.each_hash do |f|
f.puts "#{f['date']}"
end
end
I get the following error. Line 23 is the date_results.each_hash line.
so_test.rb:24:in `block (2 levels) in <main>': private method `puts' called for {"date"=>"Jun-12\n"}:Hash (NoMethodError)
from /Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/ruby-mysql-2.9.9/lib/mysql.rb:686:in `call'
from /Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/ruby-mysql-2.9.9/lib/mysql.rb:686:in `each_hash'
from so_test.rb:23:in `block in <main>'
from so_test.rb:22:in `open'
from so_test.rb:22:in `<main>'
Any advice is appreciated. Thanks.
In your nested blocks, you're overriding the f variable. Instead, try something like:
# create new xml file and insert header
open('test_trend.xml', 'w') do |f|
date_results.each_hash do |hash|
f.puts "#{hash['date']}"
end
end