What is the simplest way to do upsert with Ecto (MySQL) - mysql

Doing upsert is common in my app and I want to implement the cleanest and simple way to implement upsert.
Should I use fragments to implement native sql upsert?
Any idiomatic ecto way to do upsert?

You can use Ecto.Repo.insert_or_update/2, please note that for this to work, you will have to load existing models from the database.
model = %Post{id: 'existing_id', ...}
MyRepo.insert_or_update changeset
# => {:error, "id already exists"}
Example:
result =
case MyRepo.get(Post, id) do
nil -> %Post{id: id} # Post not found, we build one
post -> post # Post exists, using it
end
|> Post.changeset(changes)
|> MyRepo.insert_or_update
case result do
{:ok, model} -> # Inserted or updated with success
{:error, changeset} -> # Something went wrong
end

In my case insert_or_update raised an error due to the unique index constraint 🤔
What did work for me was Postgres v9.5 upsert through on_conflict parameter:
(considering unique column is called user_id)
changeset
|> MyRepo.insert(
on_conflict: :replace_all,
conflict_target: :user_id
)

If you're looking to upsert by something other than id, you can swap in get_by for get like this:
model = %User{email: "existing_or_new_email#heisenberg.net", name: "Cat", ...}
model |> User.upsert_by(:email)
# => {:found, %User{...}} || {:ok, %User{...}}
defmodule App.User do
alias App.{Repo, User}
def upsert_by(%User{} = record_struct, selector) do
case User |> Repo.get_by({selector, record_struct |> Map.get(selector)}) do
nil -> %User{} # build new user struct
user -> user # pass through existing user struct
end
|> User.changeset(record_struct |> Map.from_struct)
|> Repo.insert_or_update
end
end
On the off chance you're looking for a flexible approach that works across models and for multiple selectors (ie country + passport number), check out my hex package EctoConditionals!

Related

Seeding the Database from a CSV file in Phoenix/Elixir

When I try to run: mix run priv/repo/seeds.exs, I have a problem: (FunctionClauseError) no function clause matching in anonymous fn/1 in :elixir_compiler_1.__FILE__/1 The following arguments were given to anonymous fn/1 in :elixir_compiler_1.__FILE__/1:
This is my seeds.exs file:
alias FlightsList.Repo
alias FlightsList.Management.Flights
File.stream!("C:/Users/vukap/phx_projects/flights_list/priv/repo/flights.csv")
|> Stream.drop(1)
|> CSV.decode(headers: [:Id, :Origin, :Destination, :DepartureDate, :DepartureTime, :ArrivalDate, :ArrivalTime, :Number])
|> Enum.each(fn {:ok, map} ->
Flights.changeset(
%Flights{},
%{Id: String.to_integer(map[:Id]), Origin: map[:Origin], Destination: map[:Destination], DepartureDate: String.to_integer(map[:DepartureDate]), DepartureTime: String.to_integer(map[:DepartureTime]), ArrivalDate: String.to_integer(map[:ArrivalDate]), ArrivalTime: String.to_integer(map[:ArrivalTime]), Number: map[:Number]})
|> Repo.insert!()
end)
How can I fix it?
It’s impossible to answer precisely until you have specified CSV library you used, or at least what the error actually says after The following arguments were given to anonymous fn/1, but the issue is definitely with CSV.decode/2 returning something different from {:ok, map} your next clause expects.
To fix this and similar issues, one would add the catch-all clause to the processing and examine the outcome.
...
|> Enum.each(fn
{:ok, map} -> Flights.changeset(...)
other -> IO.inspect(other, label: "Unexpected")
end)
Check what the above would print out and fix it accordingly.
I guess you are missing a separator in CSV.decode function, here is an example of how I do it, you can call stream_csv in seed file.
def store_it(row) do
{:ok, result} = row
%Segments{
id: result.id,
name: result.name
} |> Repo.insert!
end
def stream_csv do
Path.expand("~/Project/segmments.csv")
|> File.stream!
|> CSV.decode(separator: ?;, headers: [:id, :name])
|> Enum.each(&store_it/1)
end

Dynamic embedded schema based on type

I have a json/map field in database which can actually holds 3 kinds of objects based on type field define in table
so structure is like this
field(:type, :string) => user/player/admin
field(:object, :map) => embedded schema needed here. and it can be any of the three schemas ie user/admin/player
So is this possible I can have the flexibility of mysql and beauty/support of Ecto embeds.
You will have to manually convert the json object to an embedded schema based off of the type. This can easily be accomplished with a transform.
defmodule EnrichedUser do
defmodule User do
defstruct type: nil, name: ""
def new(record), do: %User{type: record.type}
end
defmodule Player do
defstruct type: nil, name: ""
def new(record), do: %Player{type: record.type}
end
defmodule Admin do
defstruct type: nil, name: ""
def new(record), do: %Admin{type: record.type}
end
def transform(record = %MyDatabaseUser{type: type}) do
case type do
:user -> User.new(record)
:player -> Player.new(record)
:admin -> Admin.new(record)
_ -> {:error, :unhandled_type}
end
end
end
# Usage example
User |> Repo.all() |> Enum.map(&EnrichedUser.transform/1)
This has the added benefit of decoupling your application logic from your database model. Your app will now pass around an EnricedUser.X struct which will server as the contract between services instead of the Ecto schema that will change when you make database changes.

Elixir : how to display a structured data element?

I'm trying to parse a CSV file. Actually I have this code :
alias NimbleCSV.RFC4180, as: CSV
defmodule Siren do
def parseCSV do
IO.puts("Let's parse CSV file!")
stream = File.stream!("name.csv")
original_line = CSV.parse_stream(stream)
filter_line = Stream.filter(original_line, fn
["JeremyGuthrie" | _] -> true
_ -> false
end)
map = Stream.map(filter_line,
fn [name, team, position, height, weight, age] ->
%{name: name, team: team, position: position,
height: String.to_integer(height),
weight: String.to_integer(weight),
age: Float.parse(age) |> elem(0)
}
end)
end
end
According to my view I build a stream who handle each line of my name.csv file. With NimbleCSV library I parse this line and avoid the header line. Then, I filter each line to keep only the one corresponding to JeremyGuthrie. And finally I stock the line element into a structured data map. But now how to print just the name of my filter line : here JeremyGuthrie.
And I have an other question : I'm having some problems to filter my stream according to a number like an age, height or weight.
Here I apply Aleksei's advice with another code :
NimbleCSV.define(MyParser, separator: ";", escape: "\"")
defmodule Siren do
def parseCSV do
IO.puts("Let's parse CSV file!")
"ActeursEOF.csv"
|> File.stream!()
|> MyParser.parse_stream()
|> Stream.filter(fn
["RAZEL BEC" | _] -> true
["" | _] -> false
_ -> false
end)
|> Stream.map(fn [name, description, enr_competences] ->
%{name: name, description: description, enr_competences: enr_competences}
end)
|> Enum.to_list()
|> IO.inspect()
end
end
My output:
Compiling 1 file (.ex)
Let's parse CSV file!
[%{description: "GĂ©nie Civil", enr_competences: "Oui", name: "RAZEL BEC"}]
But now to close this subject I would to access and stock just the description for instance. And I don't see how to do that... And finally display this data.
Producing intermediate variables is redundant, in elixir we have Kernel.|>/2 aka pipe operator to pipe the functions’ output to the first argument of the next function.
"name.csv"
|> File.stream!()
|> CSV.parse_stream()
|> Stream.filter(fn
["JeremyGuthrie" | _] -> true
_ -> false
end)
|> Stream.map(fn
[name, team, position, height, weight, age] ->
%{name: name, team: team, position: position,
height: String.to_integer(height),
weight: String.to_integer(weight),
age: Float.parse(age) |> elem(0)
}
end)
|> Enum.to_list() # THIS
Note the last line in the chain. Streams are to be terminated to retrieve the result. Until the termination happens, it’s lazily constructed, but not evaluated at all. That makes it possible to e.g. produce and operate infinite streams.
Any greedy function from Enum module would do: Enum.take/2, or, as I pointed out above, Enum.to_list/1.
For the sake of reference, in the future, when you feel fully familiar with elixir, you might use Flow instead of Stream to parallelize mapping. For now (and for relatively small files) Stream is good enough.

In Elixir, How can I extract a lambda to a named function when the lambda is in a closure?

I have the following closure:
def get!(Item, id) do
Enum.find(
#items,
fn(item) -> item.id == id end
)
end
As I believe this looks ugly and difficult to read, I'd like to give this a name, like:
def get!(Item, id) do
defp has_target_id?(item), do: item.id = id
Enum.find(#items, has_target_id?/1)
end
Unfortunately, this results in:
== Compilation error in file lib/auction/fake_repo.ex ==
** (ArgumentError) cannot invoke defp/2 inside function/macro
(elixir) lib/kernel.ex:5238: Kernel.assert_no_function_scope/3
(elixir) lib/kernel.ex:4155: Kernel.define/4
(elixir) expanding macro: Kernel.defp/2
lib/auction/fake_repo.ex:28: Auction.FakeRepo.get!/2
Assuming it is possible, what is the correct way to do this?
The code you posted has an enormous amount of syntax errors/glitches. I would suggest you start with getting accustomed to the syntax, rather than trying to make Elixir better by inventing the things that nobody uses.
Here is the correct version that does what you wanted. The task might be accomplished with an anonymous function, although I hardly see a reason to make a perfectly looking idiomatic Elixir look ugly.
defmodule Foo do
#items [%{id: 1}, %{id: 2}, %{id: 3}]
def get!(id) do
has_target_id? = fn item -> item.id == id end
Enum.find(#items, has_target_id?)
end
end
Foo.get! 1
#⇒ %{id: 1}
Foo.get! 4
#⇒ nil
You can do this:
def get!(Item, id) do
Enum.find(
#items,
&compare_ids(&1, id)
)
end
defp compare_ids(%Item{}=item, id) do
item.id == id
end
But, that's equivalent to:
Enum.find(
#items,
fn item -> compare_ids(item, id) end
)
and may not pass your looks ugly and difficult to read test.
I was somehow under the impression Elixir supports nested functions?
Easy enough to test:
defmodule A do
def go do
def greet do
IO.puts "hello"
end
greet()
end
end
Same error:
$ iex a.ex
Erlang/OTP 20 [erts-9.2] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
** (ArgumentError) cannot invoke def/2 inside function/macro
(elixir) lib/kernel.ex:5150: Kernel.assert_no_function_scope/3
(elixir) lib/kernel.ex:3906: Kernel.define/4
(elixir) expanding macro: Kernel.def/2
a.ex:3: A.go/0
wouldn't:
defp compare_ids(item, id), do: item.id == id
be enough? Is there any advantage to including %Item{} or making
separate functions for returning both true and false conditions?
What you gain by specifying the first parameter as:
func(%Item{} = item, target_id)
is that only an Item struct will match the first parameter. Here is an example:
defmodule Item do
defstruct [:id, :name, :description]
end
defmodule Dog do
defstruct [:id, :name, :owner]
end
defmodule A do
def go(%Item{} = item), do: IO.inspect(item.id, label: "id: ")
end
In iex:
iex(1)> item = %Item{id: 1, name: "book", description: "old"}
%Item{description: "old", id: 1, name: "book"}
iex(2)> dog = %Dog{id: 1, name: "fido", owner: "joe"}
%Dog{id: 1, name: "fido", owner: "joe"}
iex(3)> A.go item
id: : 1
1
iex(4)> A.go dog
** (FunctionClauseError) no function clause matching in A.go/1
The following arguments were given to A.go/1:
# 1
%Dog{id: 1, name: "fido", owner: "joe"}
a.ex:10: A.go/1
iex(4)>
You get a function clause error if you call the function with a non-Item, and the earlier an error occurs, the better, because it makes debugging easier.
Of course, by preventing the function from accepting other structs, you make the function less general--but because it's a private function, you can't call it from outside the module anyway. On the other hand, if you wanted to call the function on both Dog and Item structs, then you could simply specify the first parameter as:
|
V
func(%{}=thing, target_id)
then both an Item and a Dog would match--but not non-maps.
What you gain by specifying the first parameter as:
|
V
func(%Item{id: id}, target_id)
is that you let erlang's pattern matching engine extract the data you need, rather than calling item.id as you would need to do with this definition:
func(%Item{}=item, target_id)
In erlang, pattern matching in a parameter list is the most efficient/convenient/stylish way to write functions. You use pattern matching to extract the data that you want to use in the function body.
Going even further, if you write the function definition like this:
same variable name
| |
V V
func(%Item{id: target_id}, target_id)
then erlang's pattern matching engine not only extracts the value for the id field from the Item struct, but also checks that the value is equal to the value of the target_id variable in the 2nd argument.
Defining multiple function clauses is a common idiom in erlang, and it is considered good style because it takes advantage of pattern matching rather than logic inside the function body. Here's an erlang example:
get_evens(List) ->
get_evens(List, []).
get_evens([Head|Tail], Results) when Head rem 2 == 0 ->
get_evens(Tail, [Head|Results]);
get_evens([Head|Tail], Results) when Head rem 2 =/= 0 ->
get_evens(Tail, Results);
get_evens([], Results) ->
lists:reverse(Results).

Seeding Phoenix Repo from json response

I'm trying to seed the database for my Phoenix application using the response from another api. I don't understand how to parse the response to create new object from it. I'm using HTTPoisin and Poison as of right now
seed.ex
alias NewsApplication.Article
HTTPoison.start
url = "rails_application.com/articles/index"
case HTTPoison.get(url) do
{:ok, %HTTPoison.Response{status_code: 200, body: body}} ->
articles = Poison.decode!(body["articles"])
Enum.fetch(articles, 1, fn(a) -> IO.puts a end)
{:ok, %HTTPoison.Response{status_code: 404}} ->
IO.puts "Not found :("
{:error, %HTTPoison.Error{reason: reason}} ->
IO.inspect reason
end
web/model/article.ex
updated
defmodule NewsApplication.Article do
use Ecto.Model
schema "articles" do
field :label, :string
field :slug, :string
field :full_path, :string
field :content_cache, :string
field :position, :integer, default: 0
field :children_count, :integer, default: 0
field :is_published, :boolean, default: false
field :is_shared, :boolean, default: false
field :featured, :boolean, default: false
field :score, :integer
timestamps
end
end
sample response
{"articles":
[{
"article":{
"id":436,
"updated":"2015-08-14T11:51:21.931Z",
"title":"Celebrating It's 50th Issue",
"numberOfViews":0,
"numberOfFavorites":2,
"imageURLs":["http://.../images/1549/original/axye.png"],
"tags":["Company News"],
"isFeatured":false,
"isPublished":true,
"published":"2015-07-28T17:00:00.000Z"
}
}]
}
It looks like you're almost there, but are missing a couple of things.
You'll want to iterate through each of the article responses using Enum.each and save these to the database like this:
Article.changeset(%Article{}, json)
|> Repo.insert
putting it all together, it'd be something like this:
Enum.each articles_json, fn(article) ->
Article.changeset(%Article{}, json)
|> Repo.insert
end
This won't handle any errors on Repo.insert if the changeset is invalid, but it'll work otherwise.
I'd recommend having a read through the Phoenix docs for models for further reference – specifically have a look at how they're structuring create actions.