Ruby on Rails - Combine several JSONs from api call - json

In my application, I need to do API-call to a client in order to get information about a product (I do it by giving product_id).
Their API only allows 1 ID at a time which would make the job many times harder.
what have in mind is to run the API-call several times and add the results to the view.
I did a small test with merging 2 JSON together, which worked fine. Example:
#products = data_1['data'] + data_2['data']
This was, of course, doing them by hand and simple. What I thought I could do was to do an each do on them and run the code, as below:
### CONTROLLER
product_id = Array(params[:product_id].split(','))
product_id.each do |id|
uri = URI.parse("https://website.com/api?product_id=#{id}")
request = Net::HTTP::Get.new(uri)
request["Accept"] = "application/json"
req_options = { use_ssl: uri.scheme == "https", }
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
data = JSON.parse(response.body)
#products = data['data']
end
### VIEW
- #products.each do |product|
%p= product.name
%p= product.date
This doesn't give me any error, but in my view, I only see details for last product_id param. Same with if I move #products = data['data'] out of the each do.
In a desperate attempt, I also tried with this with no luck:
#products = id.as_json.merge(data['data'])
Any help is appreciated!

The answer was much closer to what I had in mind. I need to run map on the variable itself, like below:
### CONTROLLER
product_id = Array(params[:product_id].split(','))
#products = product_id.map { |id|
request = Net::HTTP::Get.new(uri)
request["Accept"] = "application/json"
req_options = { use_ssl: uri.scheme == "https", }
response = Net::HTTP.start(uri.hostname, uri.port, req_options) do |http|
http.request(request)
end
data = JSON.parse(response.body)
data['data']
}
hope it helps someone :)

Related

Is there a way to fetch data in database from a Ruby-on-Rails models

I have a rails app running alongside with a rails API, there is a constant value for DAYS_LIMIT in config/initializers/constants.rb
DAYS_LIMIT = 40
DEFAULT_PRICE = 1.29
but now in the app i added an input field so that the user decide his DAYS_LIMIT.
So i want to fetch that value from the database from inside the API models.
I have placed breakpoints and can see that inside the API controller, the data is transfered from the app but not to the models.
edited as a question requested , it's a React-on-Rails app , here is the code where the new input field is save to the database (i have removed the other fields so the question look shorter)
export const saveChannel = (files) => {
return async (dispatch, getState) => {
const { channel } = getState();
const {rss_podcast_days} = channel;
const { image } = files;
const save = id ? updateChannel : createChannel;
const sub_required = subscription_required !== undefined ? subscription_required : false;
const formData = new FormData();
formData.append('channel[rss_podcast_days]', rss_podcast_days || '');
if (Object.keys(image).length) {
formData.append('channel[image]', image);
}
const channelId = await dispatch(save(formData, id));
dispatch(fetchChannel(id));
return id;
};
};
from the app controller
podcast_list = RestClient.get("#{ENV['URL_API']}/api/#{#channel.id.as_json}/podcast/list")
#podcasts = JSON.parse(podcast_list.body)
#podcasts = #podcasts.sort.reverse.to_h
this is from the API controller witch the data is transfered from the app
def index
podcasts = #channel.podcasts.published.list(params[:page], params[:items_per_page], params[:ordered_in])
render json: Podcasts::Normalizer.normalize(podcasts, #channel.station.default_podcast_price)
end
and here from the API model that i want to fetch data instead of the constants.
scope :by_days_limit, -> {with_tags.more_recent_than(Date.today - DAYS_LIMIT.days).ordered}
it should take today date minus the value (DAYS_LIMIT) from user input, but for now i get undefined local variable or method if i try to fetch directly
Bro if your class has constant like DAYS_LIMIT you can access it using that class itself for example,
class Demo
DAYS_LIMIT = 5
end
you can access that constant by Demo.DAYS_LIMIT in controller or else wherever you need it.
good luck!
ok , so i finally got it, i don't know if i should delete this thread or just tell how i did it. If it's inapropriate just tell me and i will delete this entire thread.
So here is how i did it, in the API controller i had to add my fetch so that the arguments (list) knows what i am talking about. #channel.days_limit
def index
podcasts = #channel.podcasts.published.list(params[:page], params[:items_per_page], params[:ordered_in], #channel.days_limit)
render json: Podcasts::Normalizer.normalize(podcasts, #channel.station.default_podcast_price)
end
then in the def list of the models, i added days_limit has argument
def list(page = nil, nb_items_per_page = 40, ordered_in = 'desc', days_limit)
ordered_in = ordered_in.in?(['asc', 'desc']) ? ordered_in : 'desc'
page.blank? ? by_days_limit(days_limit) : by_page(page, nb_items_per_page, ordered_in)
end
and finally in the scope of the models, i pass in the new argument
scope :by_days_limit, -> (days_limit) {with_tags.more_recent_than(Date.today - days_limit.days).ordered}
Now the user input from the app is passing to the models via the controller.

Get all rows in addition to number of all rows for pagination using json and ajax in Laravel

I want to get 10 rows from database and count them all for pagination. It works when I don't use ajax however there is a problem in encoding jason and sending it to the view.
My code in controller:
$datas = model::take(10)->get();
$dataCount = model::all()->count();
$datas = $datas->put('dataCount', $dataCount);
return json_encode($datas);
And my code in view:
.complete(function(datas)
{
data = JSON.parse(datas.responseText);
data_count = data.dataCount;
delete data['dataCount'];
count = data.length;
for(key=0; key<count; key++)
{
//do something
}
});
When I delete data['dataCount'], the data object gets undefined, but when I delete $datas = $datas->put('dataCount', $dataCount), it works however I want to use count of all rows to create a pagination.
Thanks
Probably something to do with the Eloquent collection serialization. Try not including arbitrary data to the collection. Try something like this:
$datas = model::take(10)->get();
$dataCount = model::count();
$resp['data'] = $datas->toArray();
$resp['dataCount'] = $dataCount;
return json_encode($resp);
It will return your data array in a json attribute and your count in other.

Ruby - Scraping HTML : If url does not exist then skip to next

I am currently working on a html scraper that takes a list of anime-planet url's from a text file and then loops through them, parses and stores the data in a database.
The scraper is working nicely however if I put in a large list then the chances of the url not linking to a series properly and throwing an error is quite high. I want to try make it so that IF the url does not work then it notes down the url in an array named 'error-urls' and just skips the record.
The end result being that the script finishes all working url's and returns a list of non working urls i can work with later (maybe in a text file, or just display in console).
I am currently using a rake task for this which is working quite nicely. If anyone could help me with implementing the error handling functionality it would be much appreciated. Cheers!
scrape.rake:
task :scrape => :environment do
require 'nokogiri'
require 'open-uri'
text = []
File.read("text.txt").each_line do |line|
text << line.chop
end
text.each do |series|
url = "http://www.anime-planet.com/anime/" + series
data = Nokogiri::HTML(open(url))
title = data.at_css('.theme').text
synopsis = data.at_css('.synopsis').text.strip
synopsis.slice! "Synopsis:\r\n\t\t\t\t\t"
eps = data.at_css('.type').text
year = data.at_css('.year').text
rating = data.at_css('.avgRating').text
categories = data.at_css('.categories')
genre = categories.css('li').text.to_s
image = data.at_css('#screenshots img')
imagePath = "http://www.anime-planet.com" + image['src']
anime = Series.create({:title => title, :image => imagePath, :description => synopsis, :eps => eps, :year => year, :rating => rating})
anime.tag_list = genre
anime.save()
end
end
Small example of list.txt
5-Centimeters-Per-Second
11Eyes
A-Channel
Air
Air-Gear
Aishiteru-Ze-Baby
You can use open-uri's error handling. See this for more details.
url = "http://www.anime-planet.com/anime/" + series
begin
doc = open(url)
rescue OpenURI::HTTPError => http_error
# bad status code returned
// do something here
status = http_error.io.status[0].to_i # => 3xx, 4xx, or 5xx
puts "Got a bad status code #{status}"
# http_error.message is the numeric code and text in a string
end
data = Nokogiri::HTML(doc)

rails 2.3 convert hash into mysql query

I'm trying to find out how rails converts a hash such as (This is an example please do not take this literally I threw something together to get the concept by I know this query is the same as User.find(1)):
{
:select => "users.*",
:conditions => "users.id = 1",
:order => "username"
}
Into:
SELECT users.* FROM users where users.id = 1 ORDER BY username
The closest thing I can find is ActiveRecord::Base#find_every
def find_every(options)
begin
case from = options[:from]
when Symbol
instantiate_collection(get(from, options[:params]))
when String
path = "#{from}#{query_string(options[:params])}"
instantiate_collection(format.decode(connection.get(path, headers).body) || [])
else
prefix_options, query_options = split_options(options[:params])
path = collection_path(prefix_options, query_options)
instantiate_collection( (format.decode(connection.get(path, headers).body) || []), prefix_options )
end
rescue ActiveResource::ResourceNotFound
# Swallowing ResourceNotFound exceptions and return nil - as per
# ActiveRecord.
nil
end
end
I'm unsure as to how to modify this to just return what the raw mysql statement would be.
So after a few hours of digging I came up with an answer although its not great.
class ActiveRecord::Base
def self._get_finder_options options
_get_construct_finder_sql(options)
end
private
def self._get_construct_finder_sql(options)
return (construct_finder_sql(options).inspect)
end
end
adding this as an extension gives you a publicly accessible method _get_finder_options which returns the raw sql statement.
In my case this is for a complex query to be wrapped as so
SELECT COUNT(*) as count FROM (INSERT_QUERY) as count_table
So that I could still use this with the will_paginate gem. This has only been tested in my current project so if you are trying to replicate please keep that in mind.

How do I use tmhOAuth to send an image stored in mysql DB to twitter

I am using the code below to post a status and an image to twitter. The library is tmhOAuth, the image is coming from a blob in my mysql db as is the mime type and file name.
Basically if an image is found stored in the db row then it sends to twitter using a post to "update_with_media" if no image is found then it just sends the status update alone using "update". the "update" works fine, but I can't get the update with media working. I've tried several variations from various sites including Twitter's dev site but no joy. I'd really appreciate any help with this.
if(!empty($imgid)) {
$getimg = mysql_query("SELECT filedata, filename, mime_type FROM post_images WHERE id = $imgid");
$img = #mysql_fetch_array($getimg);
$imagedata = $img['filedata'];
$imagename = $img['filename'];
$imagetype = $img['mime_type'];
$theimg = "#{$imagedata};type=$imagetype;filename={$imagename}";
$hasimg = "y";
}
if($hasimg == "y") {
$code = $connection->request( 'POST',
$connection->url('https://upload.twitter.com/1/statuses/update_with_media.json'),
array(
'media[]' => $theimg,
'status' => $thetweet),
true, // use auth
true // multipart
);
} else {
$code = $connection->request('POST',
$connection->url('1.1/statuses/update'),
array('status' => $thetweet));
}
// A response code of 200 is a success
if ($code == 200) {
print "Tweet sent";
} else {
print "Error: $code";
}
in case anyone else has a similar issue my problem was formatting. Instead of using the variable "$theimg", I replaced
'media[]' => $theimg, with 'media[]' => "$imagedata; type=$imagetype; filename=$imagename"
Hope that helps someone...
Peace!