Hi I have Person schema in elixir like this
[
%Texting.Contact.Person{
__meta__: #Ecto.Schema.Metadata<:loaded, "people">,
email: nil,
name: "John",
phone_number: "13334445555",
phonebook: #Ecto.Association.NotLoaded<association :phonebook is not loaded>,
phonebook_id: 60,
previous_phonebook_id: 60,
subscribed: true,
updated_at: ~N[2018-09-22 14:36:04.788163],
user: #Ecto.Association.NotLoaded<association :user is not loaded>,
user_id: 54
},
%Texting.Contact.Person{
__meta__: #Ecto.Schema.Metadata<:loaded, "people">,
email: nil,
name: "Rhee",
phone_number: "14443335555",
phonebook: #Ecto.Association.NotLoaded<association :phonebook is not loaded>,
phonebook_id: 60,
previous_phonebook_id: 60,
subscribed: true,
updated_at: ~N[2018-09-22 14:36:13.671479],
user: #Ecto.Association.NotLoaded<association :user is not loaded>,
user_id: 54
}
]
And I want to save this to csv file format.So I did this
def write!(people) do
file = File.open("contacts.csv", [:write, :utf8])
people
|> Enum.map(&Map.from_struct(&1))
|> Enum.map(&CSV.encode(&1, headers: [:name, :phone_number]))
|> Enum.map(&IO.write(file, &1))
end
but I got an error like this
** (Protocol.UndefinedError) protocol String.Chars not implemented for #Function<62.51129937/2 in Stream.transform/3>. This protocol is implemented for: Atom, BitString, Date, DateTime, Decimal, Ecto.Date, Ecto.DateTime, Ecto.Time, Float, Floki.Selector, Floki.Selector.AttributeSelector, Floki.Selector.Combinator, Floki.Selector.Functional, Floki.Selector.PseudoClass, Integer, List, NaiveDateTime, Postgrex.Copy, Postgrex.Query, Postgrex.Stream, Time, URI, Version, Version.Requirement
(elixir) /home/ubuntu/bob/tmp/0a92cc555e2418d1b56e3b10e5321a85/elixir/lib/elixir/lib/string/chars.ex:3: String.Chars.impl_for!/1
(elixir) /home/ubuntu/bob/tmp/0a92cc555e2418d1b56e3b10e5321a85/elixir/lib/elixir/lib/string/chars.ex:22: String.Chars.to_string/1
(elixir) lib/io.ex:553: IO.write/2
(elixir) lib/enum.ex:1314: Enum."-map/2-lists^map/1-0-"/2
What I want to do is save only name and phone_number field to csv file format.
How can I do this?
According to the documentation, CSV.encode/2 takes a list of list of strings and doesn't have the option you tried to use.
Here's how I'd do it:
def write!(people) do
people
|> Stream.map(&[&1.name, &1.phone_number])
|> CSV.encode()
|> Enum.into(File.stream!("contacts.csv"))
end
The Enum.map creates a list of list of strings which is then encoded and streamed into contacts.csv.
To add a header line at the top, you can use Stream.concat/2 to prepend the header line:
def write!(people) do
[["name", "phone_number"]]
|> Stream.concat(people |> Stream.map(&[&1.name, &1.phone_number]))
|> CSV.encode()
|> Enum.into(File.stream!("contacts.csv"))
end
Related
I have a json/map field in database which can actually holds 3 kinds of objects based on type field define in table
so structure is like this
field(:type, :string) => user/player/admin
field(:object, :map) => embedded schema needed here. and it can be any of the three schemas ie user/admin/player
So is this possible I can have the flexibility of mysql and beauty/support of Ecto embeds.
You will have to manually convert the json object to an embedded schema based off of the type. This can easily be accomplished with a transform.
defmodule EnrichedUser do
defmodule User do
defstruct type: nil, name: ""
def new(record), do: %User{type: record.type}
end
defmodule Player do
defstruct type: nil, name: ""
def new(record), do: %Player{type: record.type}
end
defmodule Admin do
defstruct type: nil, name: ""
def new(record), do: %Admin{type: record.type}
end
def transform(record = %MyDatabaseUser{type: type}) do
case type do
:user -> User.new(record)
:player -> Player.new(record)
:admin -> Admin.new(record)
_ -> {:error, :unhandled_type}
end
end
end
# Usage example
User |> Repo.all() |> Enum.map(&EnrichedUser.transform/1)
This has the added benefit of decoupling your application logic from your database model. Your app will now pass around an EnricedUser.X struct which will server as the contract between services instead of the Ecto schema that will change when you make database changes.
I'm trying to parse a CSV file. Actually I have this code :
alias NimbleCSV.RFC4180, as: CSV
defmodule Siren do
def parseCSV do
IO.puts("Let's parse CSV file!")
stream = File.stream!("name.csv")
original_line = CSV.parse_stream(stream)
filter_line = Stream.filter(original_line, fn
["JeremyGuthrie" | _] -> true
_ -> false
end)
map = Stream.map(filter_line,
fn [name, team, position, height, weight, age] ->
%{name: name, team: team, position: position,
height: String.to_integer(height),
weight: String.to_integer(weight),
age: Float.parse(age) |> elem(0)
}
end)
end
end
According to my view I build a stream who handle each line of my name.csv file. With NimbleCSV library I parse this line and avoid the header line. Then, I filter each line to keep only the one corresponding to JeremyGuthrie. And finally I stock the line element into a structured data map. But now how to print just the name of my filter line : here JeremyGuthrie.
And I have an other question : I'm having some problems to filter my stream according to a number like an age, height or weight.
Here I apply Aleksei's advice with another code :
NimbleCSV.define(MyParser, separator: ";", escape: "\"")
defmodule Siren do
def parseCSV do
IO.puts("Let's parse CSV file!")
"ActeursEOF.csv"
|> File.stream!()
|> MyParser.parse_stream()
|> Stream.filter(fn
["RAZEL BEC" | _] -> true
["" | _] -> false
_ -> false
end)
|> Stream.map(fn [name, description, enr_competences] ->
%{name: name, description: description, enr_competences: enr_competences}
end)
|> Enum.to_list()
|> IO.inspect()
end
end
My output:
Compiling 1 file (.ex)
Let's parse CSV file!
[%{description: "Génie Civil", enr_competences: "Oui", name: "RAZEL BEC"}]
But now to close this subject I would to access and stock just the description for instance. And I don't see how to do that... And finally display this data.
Producing intermediate variables is redundant, in elixir we have Kernel.|>/2 aka pipe operator to pipe the functions’ output to the first argument of the next function.
"name.csv"
|> File.stream!()
|> CSV.parse_stream()
|> Stream.filter(fn
["JeremyGuthrie" | _] -> true
_ -> false
end)
|> Stream.map(fn
[name, team, position, height, weight, age] ->
%{name: name, team: team, position: position,
height: String.to_integer(height),
weight: String.to_integer(weight),
age: Float.parse(age) |> elem(0)
}
end)
|> Enum.to_list() # THIS
Note the last line in the chain. Streams are to be terminated to retrieve the result. Until the termination happens, it’s lazily constructed, but not evaluated at all. That makes it possible to e.g. produce and operate infinite streams.
Any greedy function from Enum module would do: Enum.take/2, or, as I pointed out above, Enum.to_list/1.
For the sake of reference, in the future, when you feel fully familiar with elixir, you might use Flow instead of Stream to parallelize mapping. For now (and for relatively small files) Stream is good enough.
I'm getting this string as query result from my database:
"%Sample.Struct{list: [], total: \"0.00\", day: 6, id: \"8vfts6\"}"
Is there any way to convert this one back to map?
I'm getting this error decoding it with poison
** (Poison.SyntaxError) Unexpected token: %
(poison) lib/poison/parser.ex:56: Poison.Parser.parse!/2
(poison) lib/poison.ex:83: Poison.decode!/2
I can't fix the way data is being added to database, i must find a proper way for a key/value route to easily retrive data from that. (this is just a sample for a more complex result)
As it was mentioned in comments, you should not use Code.eval_string. But, there is a way to safely convert your code to Elixir struct, using Code module:
ex(1)> encoded = "%Sample.Struct{list: [], total: \"0.00\", day: 6, id: \"8vfts6\"}"
"%Sample.Struct{list: [], total: \"0.00\", day: 6, id: \"8vfts6\"}"
First, get the AST from the string, but use the pattern matching to ensure it is a struct you are looking for ({:__aliases__, _, [:Sample, :Struct]}). All other (potentially malicious) code will fail this match:
iex(2)> {:ok, {:%, _, [{:__aliases__, _, [:Sample, :Struct]}, {:%{}, _, keymap}]} = ast} = Code.string_to_quoted(encoded)
{:ok,
{:%, [line: 1],
[{:__aliases__, [line: 1], [:Sample, :Struct]},
{:%{}, [line: 1], [list: [], total: "0.00", day: 6, id: "8vfts6"]}]}}
Here you have the full ast for you struct, and the keymap. You may now be tempted to use eval_quoted with the AST, to get the struct you needed:
iex(3)> {struct, _} = Code.eval_quoted(ast)
{%Sample.Struct{day: 6, id: "8vfts6", list: [], total: "0.00"}, []}
iex(4)> struct
%Sample.Struct{day: 6, id: "8vfts6", list: [], total: "0.00"}
But it is still not safe! Someone may put a function causing side effect into the string, like "%Sample.Struct{list: IO.puts \"Something\"}", which will be executed during the evaluation. So you will need to check the keymap firsts, if it contain safe data.
Or you may just use keymap directly, without evaluating anyting:
iex(5)> struct(Sample.Struct, keymap)
%Sample.Struct{day: 6, id: "8vfts6", list: [], total: "0.00"}
I'm trying to seed the database for my Phoenix application using the response from another api. I don't understand how to parse the response to create new object from it. I'm using HTTPoisin and Poison as of right now
seed.ex
alias NewsApplication.Article
HTTPoison.start
url = "rails_application.com/articles/index"
case HTTPoison.get(url) do
{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
articles = Poison.decode!(body["articles"])
Enum.fetch(articles, 1, fn(a) -> IO.puts a end)
{:ok, %HTTPoison.Response{status_code: 404}} ->
IO.puts "Not found :("
{:error, %HTTPoison.Error{reason: reason}} ->
IO.inspect reason
end
web/model/article.ex
updated
defmodule NewsApplication.Article do
use Ecto.Model
schema "articles" do
field :label, :string
field :slug, :string
field :full_path, :string
field :content_cache, :string
field :position, :integer, default: 0
field :children_count, :integer, default: 0
field :is_published, :boolean, default: false
field :is_shared, :boolean, default: false
field :featured, :boolean, default: false
field :score, :integer
timestamps
end
end
sample response
{"articles":
[{
"article":{
"id":436,
"updated":"2015-08-14T11:51:21.931Z",
"title":"Celebrating It's 50th Issue",
"numberOfViews":0,
"numberOfFavorites":2,
"imageURLs":["http://.../images/1549/original/axye.png"],
"tags":["Company News"],
"isFeatured":false,
"isPublished":true,
"published":"2015-07-28T17:00:00.000Z"
}
}]
}
It looks like you're almost there, but are missing a couple of things.
You'll want to iterate through each of the article responses using Enum.each and save these to the database like this:
Article.changeset(%Article{}, json)
|> Repo.insert
putting it all together, it'd be something like this:
Enum.each articles_json, fn(article) ->
Article.changeset(%Article{}, json)
|> Repo.insert
end
This won't handle any errors on Repo.insert if the changeset is invalid, but it'll work otherwise.
I'd recommend having a read through the Phoenix docs for models for further reference – specifically have a look at how they're structuring create actions.
I am using Postgres' json data type but want to do a query/ordering with data that is nested within the json.
I want to order or query with .where on the json data type. For example, I want to query for users that have a follower count > 500 or I want to order by follower or following count.
Thanks!
Example:
model User
data: {
"photos"=>[
{"type"=>"facebook", "type_id"=>"facebook", "type_name"=>"Facebook", "url"=>"facebook.com"}
],
"social_profiles"=>[
{"type"=>"vimeo", "type_id"=>"vimeo", "type_name"=>"Vimeo", "url"=>"http://vimeo.com/", "username"=>"v", "id"=>"1"},
{"bio"=>"I am not a person, but a series of plants", "followers"=>1500, "following"=>240, "type"=>"twitter", "type_id"=>"twitter", "type_name"=>"Twitter", "url"=>"http://www.twitter.com/", "username"=>"123", "id"=>"123"}
]
}
For any who stumbles upon this. I have come up with a list of queries using ActiveRecord and Postgres' JSON data type. Feel free to edit this to make it more clear.
Documentation to the JSON operators used below: https://www.postgresql.org/docs/current/functions-json.html.
# Sort based on the Hstore data:
Post.order("data->'hello' DESC")
=> #<ActiveRecord::Relation [
#<Post id: 4, data: {"hi"=>"23", "hello"=>"22"}>,
#<Post id: 3, data: {"hi"=>"13", "hello"=>"21"}>,
#<Post id: 2, data: {"hi"=>"3", "hello"=>"2"}>,
#<Post id: 1, data: {"hi"=>"2", "hello"=>"1"}>]>
# Where inside a JSON object:
Record.where("data ->> 'likelihood' = '0.89'")
# Example json object:
r.column_data
=> {"data1"=>[1, 2, 3],
"data2"=>"data2-3",
"array"=>[{"hello"=>1}, {"hi"=>2}],
"nest"=>{"nest1"=>"yes"}}
# Nested search:
Record.where("column_data -> 'nest' ->> 'nest1' = 'yes' ")
# Search within array:
Record.where("column_data #>> '{data1,1}' = '2' ")
# Search within a value that's an array:
Record.where("column_data #> '{array,0}' ->> 'hello' = '1' ")
# this only find for one element of the array.
# All elements:
Record.where("column_data ->> 'array' LIKE '%hello%' ") # bad
Record.where("column_data ->> 'array' LIKE ?", "%hello%") # good
According to this http://edgeguides.rubyonrails.org/active_record_postgresql.html#json
there's a difference in using -> and ->>:
# db/migrate/20131220144913_create_events.rb
create_table :events do |t|
t.json 'payload'
end
# app/models/event.rb
class Event < ActiveRecord::Base
end
# Usage
Event.create(payload: { kind: "user_renamed", change: ["jack", "john"]})
event = Event.first
event.payload # => {"kind"=>"user_renamed", "change"=>["jack", "john"]}
## Query based on JSON document
# The -> operator returns the original JSON type (which might be an object), whereas ->> returns text
Event.where("payload->>'kind' = ?", "user_renamed")
So you should try Record.where("data ->> 'status' = 200 ") or the operator that suits your query (http://www.postgresql.org/docs/current/static/functions-json.html).
Your question doesn't seem to correspond to the data you've shown, but if your table is named users and data is a field in that table with JSON like {count:123}, then the query
SELECT * WHERE data->'count' > 500 FROM users
will work. Take a look at your database schema to make sure you understand the layout and check that the query works before complicating it with Rails conventions.
JSON filtering in Rails
Event.create( payload: [{ "name": 'Jack', "age": 12 },
{ "name": 'John', "age": 13 },
{ "name": 'Dohn', "age": 24 }]
Event.where('payload #> ?', '[{"age": 12}]')
#You can also filter by name key
Event.where('payload #> ?', '[{"name": "John"}]')
#You can also filter by {"name":"Jack", "age":12}
Event.where('payload #> ?', {"name":"Jack", "age":12}.to_json)
You can find more about this here