Problems Parsing a Json File in Ruby - json

I'm pretty new to programming ruby and using json packages, I think I'm facing a typical noob-error, but I just cant find my mistake for two days now.. I've started like this:
require 'discogs-wrapper'
require 'json'
aw = Discogs::Wrapper.new("my_application", user_token: "my_user_token")
inv = aw.get_user_inventory('my_user_name', :per_page => 1, :page => 1)
p=JSON.parse(inv)
What I recieve is this:
C:/Ruby24-x64/lib/ruby/2.4.0/json/common.rb:156:in `initialize': no implicit conversion of Hashie::Mash into String (TypeError)
from C:/Ruby24-x64/lib/ruby/2.4.0/json/common.rb:156:in `new'
from C:/Ruby24-x64/lib/ruby/2.4.0/json/common.rb:156:in `parse'
from C:/Users/rtuz2th/.RubyMine2017.3/config/scratches/scratch.rb:6:in `<top (required)>'
from -e:1:in `load'
from -e:1:in `<main>'
I've already been searching your forum and I've learned that this error probably means that I do not have a correct JSON-file (although it comes straight from the discogs-api, I basically thought these guys know what they do). So what I tried next was
require 'discogs-wrapper'
require 'json'
aw = Discogs::Wrapper.new("my_application", user_token: "my_user_token")
inv = aw.get_user_inventory('my_user_name', :per_page => 1, :page => 1)
inv = inv.to_json
p=JSON.parse(inv)
So at least, my code runs without any errors now. But when I try to refer to any information in inv, I just get empty responses. inv (without the .to_json looks like this:
{"pagination"=>
{"per_page"=>1,
"items"=>13692,
"page"=>1,
"urls"=>
{"last"=>
"URL containing tokens",
"next"=>
"URL containing tokens"},
"pages"=>13692},
"listings"=>
[{"status"=>"Draft",
"original_price"=>
{"curr_abbr"=>"EUR",
"formatted"=>"\u20AC10.46",
"value"=>10.46,
"curr_id"=>3},
"weight"=>230.0,
"original_shipping_price"=>
{"curr_abbr"=>"EUR",
"formatted"=>"\u20AC5.90",
"value"=>5.9,
"curr_id"=>3},
"price"=>{"currency"=>"EUR", "value"=>10.46},
"allow_offers"=>true,
"uri"=>"https://www.discogs.com/sell/item/437965910",
"sleeve_condition"=>"Very Good Plus (VG+)",
"format_quantity"=>1,
"id"=>437965910,
"shipping_price"=>{"currency"=>"EUR", "value"=>5.9},
"posted"=>"2017-02-07T23:43:01-08:00",
"ships_from"=>"Germany",
"in_cart"=>false,
"comments"=>"",
"seller"=>
{"username"=>"zweischeiben.de",
"stats"=>{"rating"=>"100.0", "total"=>143, "stars"=>5.0},
"uid"=>3359767,
"url"=>"https://api.discogs.com/users/zweischeiben.de",
"html_url"=>"https://www.discogs.com/user/zweischeiben.de",
"shipping"=>
"(Lots of information about shipping, cut just cut it out)"
"payment"=>"Bank Transfer, PayPal",
"avatar_url"=>
"(cut out url)",
"resource_url"=>"https://api.discogs.com/users/zweischeiben.de",
"id"=>3359767},
"condition"=>"Near Mint (NM or M-)",
"release"=>
{"thumbnail"=>
"(cut out url)",
"description"=>"Steamhammer - Mountains (LP, Album, RE)",
"artist"=>"Steamhammer",
"format"=>"LP, Album, RE",
"resource_url"=>"https://api.discogs.com/releases/7303333",
"title"=>"Mountains",
"year"=>0,
"id"=>7303333,
"catalog_number"=>"201.006, 0201.006"},
"resource_url"=>"https://api.discogs.com/marketplace/listings/437965910",
"audio"=>false,
"external_id"=>"3, 16",
"location"=>"T52574"}]}
But if I now try to refer to eg the location:
require 'discogs-wrapper'
require 'json'
aw = Discogs::Wrapper.new("my_application", user_token: "my_token")
inv=aw.get_user_inventory('my_username', :per_page => 1, :page => 1)
inv=inv.to_json
p=JSON.parse(inv)
puts p[location]
I'm just getting an error like this
C:\Ruby24-x64\bin\ruby.exe -e $stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) C:/Users/rtuz2th/.RubyMine2017.3/config/scratches/scratch.rb
C:/Users/rtuz2th/.RubyMine2017.3/config/scratches/scratch.rb:9:in `<top (required)>': undefined local variable or method `location' for main:Object (NameError)
from -e:1:in `load'
from -e:1:in `<main>'
Process finished with exit code 1
or an empty response. I'm trying to solve this for two days now, but I'm absolutely out of ideas. Thanks for your help in advance!

inv = inv.to_json
p = JSON.parse(inv)
The above is a noop. Use inv as is, it’s a hash already.
puts p[location]
You cannot just lookup whatever key located deeply in the tree hierarchy or you hash. You hash has stings as keys, it has a two top-level elements, 'pagination' and 'listings', the latter is an array, and each element of this array is a hash having location key.
While you were cutting the long value, you dropped the ending comma, making me unable to copy-paste the input and provide the exact answer.

Related

JSON to Hash in Ruby and vice-versa using Files - Parser Error

I am trying to save data from a Hash to a file. I convert it to JSON and dump it into the file.
When I try to parse back from file to hash I get JSON::ParserError
Code to convert Hash to JSON file: (works fine)
user = {:email => "cumber#cc.cc", :passwrd => "hardPASSw0r|)"}
student_file = File.open("students.txt", "a+") do |f|
f.write JSON.dump(user)
end
After adding a few values one by one to the file it looks something like this:
{"email":"test1#gmail.com","passwrd":"qwert123"}{"email":"test3#gmail.com","passwrd":"qwert12345"}{"email":"cumber#cc.cc","passwrd":"hardPASSw0r|)"}
I tried the following code to convert back to Hash but it doesn't work:
file = File.read('students.txt')
data_hash = JSON.parse(file)
I get
System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/json/common.rb:155:in `parse': 757: unexpected token at '{"email":"test3#gmail.com","passwrd":"qwert12345"}{"email":"cumber#cc.cc","passwrd":"hardPASSw0r|)"}' (JSON::ParserError)
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/json/common.rb:155:in `parse'
from hash_json.rb:25:in `<main>'
My goal is to be able to add and remove values from the file.
How do I fix this, where was my mistake? Thank you.
This should work:
https://repl.it/EXGl/0
# as adviced by #EricDuminil, on some envs you need to include 'json' too
require 'json'
user = {:email => "cumber#cc.cc", :passwrd => "hardPASSw0r|)"}
student_file = File.open("students.txt", "w") do |f|
f.write(user.to_json)
end
file = File.read('students.txt')
puts "saved content is: #{JSON.parse(file)}"
p.s. hope that this is only an example, never store passwords in plain-text! NEVER ;-)

Rails4, MySql, mysql2 encoding issue

I have a project where Rails 4, MySql 5.1, gem mysql2 are used.
This stack worked fine until today. I've added two fields in a table (added corresponding migration) and if I save data to the table, the data in this field will look ugly, like -%D0%9D%D0%BE%D0%B4%D0%B0%D1%80%D0%B8
Here is Rails log:
This is the data from a request. All looks fine, 'name' and 'comment' are in Russian.
Parameters: {"{\"user_id\":\"1\",\"name\":\"АБВ\",\"email\":\"mike#outofcloud.ru\",\"answer\":\"5\",\"comment\":\"цувцув\"}"=>nil}
But in the SQL statement the 'name' parameter changed:
SQL (23.3ms) INSERT INTO "feedbacks" ("answer", "code", "comment", "created_at", "email", "name", "updated_at", "user_id") VALUES ($1, $2, $3, $4, $5, $6, $7, $8) RETURNING "id" [["answer", 5], ["code", 1], ["comment", "цувцув"], ["created_at", "2015-05-13 08:54:22.047321"], ["email", "example#example.ru"], ["name", **"%D0%90%D0%91%D0%92"**], ["updated_at", "2015-05-13 08:54:22.047321"], ["user_id", "1"]]
'comment' field was added before my migrations, it works as expected. It look great in the DB and in the app.
'name' field have been added recently and its doesn't work as expected.
Can someone help with the issue?
The string that you're showing is the result of using URI.encode on "АБВ" - see:
[4] pry(main)> URI.encode "АБВ"
=> "%D0%90%D0%91%D0%92"
Rails doesn't do this by itself. You must have some code somewhere else (or, hopefully not, a gem) that's performing that transformation.
If you don't know where that is then please add the code from your controllers showing any before_action methods, or other transformations. Also the code from your model if you're doing anything before_save or any custom attribute setters, eg. def name=

undefined method `+#' for nil:NilClass rails after moving to mysql

I moved the database to mysql which is on VM. After moving database I started getting this error when I am trying to access or saving data to database but some models updation are working fine.
The model I am accessing has some arithmetic computations which I feel is the reason for this issue.
Error is
undefined method `+#' for nil:NilClass
I tried from rails console as well and I am getting same error with this command
Settings.all
I guess this is part of that method method
if params[:lat] and params[:lon] and params[:rad] and !params[:lat].empty? and
!params[:lon].empty? and !params[:rad].empty?
lat = params[:lat].to_f
lon = params[:lon].to_f
rad = params[:rad].to_f
ear = 6371.00
min_lat = (lat) - (rad/ear)/180.0*Math::PI;
max_lat = (lat) + (rad/ear)/180.0*Math::PI;
min_lon = (lon) - (rad/ear/Math.cos(lon*Math::PI/180.0))/180.0*Math::PI;
max_lon = (lon) + (rad/ear/Math.cos(lon*Math::PI/180.0))/180.0*Math::PI;
#settings = #settings.includes(:settings_location)
.where("settings_locations.latitude > :min_lat AND " \
"settings_locations.latitude < :max_lat AND " \
"settings_locations.longitude > :min_lon AND " \
"settings_locations.longitude < :max_lon", {
min_lat: min_lat,
max_lat: max_lat,
min_lon: min_lon,
max_lon: max_lon});
stack trace
Started POST "/settings.json" for 127.0.0.1 at 2014-04-24 13:52:53 +0200
NoMethodError - undefined method `+#' for nil:NilClass:
app/models/settings.rb:54:in `<top (required)>'
activesupport (4.0.2) lib/active_support/dependencies.rb:424:in `block in load_file'
activesupport (4.0.2) lib/active_support/dependencies.rb:616:in `new_constants_in'
activesupport (4.0.2) lib/active_support/dependencies.rb:423:in `load_file'
activesupport (4.0.2) lib/active_support/dependencies.rb:324:in `require_or_load'
activesupport (4.0.2) lib/active_support/dependencies.rb:463:in `load_missing_constant'
setting.rb:
class Settings < ActiveRecord::Base
has_one :settings_location
def to_builder
Jbuilder.new do |json|
json.location settings_location.to_builder
end
end //this line is 54
Per the comments, there seem to be only a handful of cases where you can reproduce the same error message in IRB. The first is:
foo, bar = nil, nil
foo.send("+#", bar) # undefined method `+#' for nil:NilClass
The second is:
foo = nil
+foo # undefined method `+#' for nil:NilClass
In an eval context, the latter might more look like:
foo, bar = nil, nil
eval("foo = +#{bar.inspect}") # undefined method `+#' for nil:NilClass
My personal guess is that your problem is something along the lines of the latter. Not with eval, but rather with one of its plethora of Ruby variations such as define_method. The actual call is very likely somewhere in Rail's internals and related to some uninitialized variable — perhaps MySQL is returning null where SQLite was returning something due to a recent schema change that you didn't think about?
Personally, I'd try to narrow it down step by step:
1. Comment out:
json.location settings_location.to_builder
If the error is gone, then:
2.a) Replace it with:
builder = settings_location.to_builder
json.location builder
If the error message is still the same, then:
2.b) Try
Jbuilder.new
… without a block.
The point is, you want to narrow things down to the specific method that is causing the error, and then dive into that method's code to see what is nil and shouldn't be. (Perhaps the block is missing some kind of return value?)

List Files: Backend Error (No error code) Google Drive SDK

we've built a web app that automatically indexes and saves email attachments directly to Google Drive based on user-defined business rules. We're seeing failures when syncing some Google Drive accounts and can't figure out what the problem is.
This is the error:
List Files Error: Backend Error
/mnt/opt/openera/current/app/models/cloud_account/google_drive.rb:299:in 'list_files'
/mnt/opt/openera/current/app/models/cloud_account/google_drive.rb:220:in 'do_discover_files'
/mnt/opt/openera/current/app/models/cloud_account/base.rb:304:in 'discover_files'
/mnt/opt/openera/current/app/workers/worker/file_synchronizer.rb:10:in 'perform'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:45:in 'block (3 levels) in
process'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:109:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:109:in 'block in in
voke'
/mnt/opt/openera/current/lib/sidekiq_repeat_jobs/middleware/server/repeat_jobs.rb:36:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/timeout.rb:14:in 'cal
l'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/active_record.rb:6:in
'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/retry_jobs.rb:49:in '
call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/logging.rb:11:in 'blo
ck in call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/logging.rb:22:in 'with_context'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/server/logging.rb:7:in 'call
'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:111:in 'block in in
voke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:114:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/middleware/chain.rb:114:in 'invoke'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:44:in 'block (2 levels) in
process'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:80:in 'stats'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/sidekiq-2.6.5/lib/sidekiq/processor.rb:43:in 'block in process'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in 'public_send'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/calls.rb:23:in 'dispatch'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/future.rb:18:in 'block in initiali
ze'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in 'call'
/mnt/opt/openera/current/vendor/bundle/ruby/1.9.1/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in 'block in c
reate'
UPDATE: Here is the actual code from our senior dev.
dev here (ffoeg) adding actual code invoked:
def list_files
pageToken = nil
params = {'maxResults' => 65535, 'q' => 'trashed = false'}
files = []
drive = session.discovered_api("drive", "v2")
while true do
call_params = pageToken.blank? ? params : params.merge('pageToken' => pageToken)
result = session.execute(api_method: drive.files.list, parameters: call_params)
if result.status == 200
files += result.data.items
break if result.data['nextPageToken'].blank?
pageToken = result.data['nextPageToken']
else
raise "List Files Error: #{result.data['error']['message']}"
end
end
files
end
UPDATING AGAIN:
We updated our code to get the exact error - here it is:
List Files Error: {"errors"=>[{"domain"=>"global", "reason"=>"backendError",
"message"=>"Backend Error"}], "code"=>503, "message"=>"Backend Error"}
The problem is the value of maxResults. It is far too large and will cause the request to time out. Unfortunately, a "safe" value is very small, about ~200 results per page. You can go larger, but results will vary based on the user. Users with smaller corpuses will probably work fine with larger page sizes, but people with larger corpuses appear to be more sensitive. Well, at least based on some very quick/informal testing on a few of my accounts.
FWIW, I remember things working better with larger page sizes. Will check if anything changed that might constrain things more than expected.

How to save database results to text file in Ruby

I am trying to take the results of a query to a db and save them to a csv file. Here is my current Ruby script.
#! /usr/bin/ruby
require 'rubygems'
require 'mysql'
require 'date'
# need mysql queries here / use mysql2 calls
db_con = Mysql.new("localhost", "root", "", "msd")
date_results = db_con.query("SELECT CONCAT(CONVERT(date_format(dd.date, '%b-%e'),char),'\\n') AS date
FROM msd.date_dim dd LEFT OUTER JOIN msd.results cs
ON dd.id = cs.date_id
GROUP BY
dd.date")
# create new xml file and insert header
open('test_trend.xml', 'w') do |f|
date_results.each_hash do |f|
f.puts "#{f['date']}"
end
end
I get the following error. Line 23 is the date_results.each_hash line.
so_test.rb:24:in `block (2 levels) in <main>': private method `puts' called for {"date"=>"Jun-12\n"}:Hash (NoMethodError)
from /Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/ruby-mysql-2.9.9/lib/mysql.rb:686:in `call'
from /Users/pierce/.rbenv/versions/1.9.2-p290/lib/ruby/gems/1.9.1/gems/ruby-mysql-2.9.9/lib/mysql.rb:686:in `each_hash'
from so_test.rb:23:in `block in <main>'
from so_test.rb:22:in `open'
from so_test.rb:22:in `<main>'
Any advice is appreciated. Thanks.
In your nested blocks, you're overriding the f variable. Instead, try something like:
# create new xml file and insert header
open('test_trend.xml', 'w') do |f|
date_results.each_hash do |hash|
f.puts "#{hash['date']}"
end
end