I'm trying to parse a CSV file. Actually I have this code :
alias NimbleCSV.RFC4180, as: CSV
defmodule Siren do
def parseCSV do
IO.puts("Let's parse CSV file!")
stream = File.stream!("name.csv")
original_line = CSV.parse_stream(stream)
filter_line = Stream.filter(original_line, fn
["JeremyGuthrie" | _] -> true
_ -> false
end)
map = Stream.map(filter_line,
fn [name, team, position, height, weight, age] ->
%{name: name, team: team, position: position,
height: String.to_integer(height),
weight: String.to_integer(weight),
age: Float.parse(age) |> elem(0)
}
end)
end
end
According to my view I build a stream who handle each line of my name.csv file. With NimbleCSV library I parse this line and avoid the header line. Then, I filter each line to keep only the one corresponding to JeremyGuthrie. And finally I stock the line element into a structured data map. But now how to print just the name of my filter line : here JeremyGuthrie.
And I have an other question : I'm having some problems to filter my stream according to a number like an age, height or weight.
Here I apply Aleksei's advice with another code :
NimbleCSV.define(MyParser, separator: ";", escape: "\"")
defmodule Siren do
def parseCSV do
IO.puts("Let's parse CSV file!")
"ActeursEOF.csv"
|> File.stream!()
|> MyParser.parse_stream()
|> Stream.filter(fn
["RAZEL BEC" | _] -> true
["" | _] -> false
_ -> false
end)
|> Stream.map(fn [name, description, enr_competences] ->
%{name: name, description: description, enr_competences: enr_competences}
end)
|> Enum.to_list()
|> IO.inspect()
end
end
My output:
Compiling 1 file (.ex)
Let's parse CSV file!
[%{description: "Génie Civil", enr_competences: "Oui", name: "RAZEL BEC"}]
But now to close this subject I would to access and stock just the description for instance. And I don't see how to do that... And finally display this data.
Producing intermediate variables is redundant, in elixir we have Kernel.|>/2 aka pipe operator to pipe the functions’ output to the first argument of the next function.
"name.csv"
|> File.stream!()
|> CSV.parse_stream()
|> Stream.filter(fn
["JeremyGuthrie" | _] -> true
_ -> false
end)
|> Stream.map(fn
[name, team, position, height, weight, age] ->
%{name: name, team: team, position: position,
height: String.to_integer(height),
weight: String.to_integer(weight),
age: Float.parse(age) |> elem(0)
}
end)
|> Enum.to_list() # THIS
Note the last line in the chain. Streams are to be terminated to retrieve the result. Until the termination happens, it’s lazily constructed, but not evaluated at all. That makes it possible to e.g. produce and operate infinite streams.
Any greedy function from Enum module would do: Enum.take/2, or, as I pointed out above, Enum.to_list/1.
For the sake of reference, in the future, when you feel fully familiar with elixir, you might use Flow instead of Stream to parallelize mapping. For now (and for relatively small files) Stream is good enough.
Hello I'm a beginner in Elixir and I want to parse and stock a CSV file in an Elixir object.
But it's display that:
** (FunctionClauseError) no function clause matching in anonymous fn/1 in Siren.parseCSV/0
The following arguments were given to anonymous fn/1 in Siren.parseCSV/0:
# 1
["41", "5", "59", "N", "80", "39", "0", "W", "Youngstown", "OH"]
anonymous fn/1 in Siren.parseCSV/0
(elixir 1.10.3) lib/stream.ex:482: anonymous fn/4 in Stream.filter/2
(elixir 1.10.3) lib/stream.ex:1449: Stream.do_element_resource/6
(elixir 1.10.3) lib/stream.ex:1609: Enumerable.Stream.do_each/4
(elixir 1.10.3) lib/enum.ex:959: Enum.find/3
(mix 1.10.3) lib/mix/task.ex:330: Mix.Task.run_task/3
(mix 1.10.3) lib/mix/cli.ex:82: Mix.CLI.run_task/2
Here my code:
defmodule Siren do
def parseCSV do
IO.puts("Let's parse CSV file...")
File.stream!("../name.csv")
|> Stream.map(&String.trim(&1))
|> Stream.map(&String.split(&1, ","))
|> Stream.filter(fn
["LatD" | _] -> false
end)
|> Enum.find(fn State -> String
[LatD, LatM, LatS, NS, LonD, LonM, LonS, EW, City, State] ->
IO.puts("find -> #{State}")
true
end)
end
end
And the csv file:
LatD,LatM,LatS,NS,LonD,LonM,LonS,EW,City,State
41,5,59,N,80,39,0,W,Youngstown,OH
42,52,48,N,97,23,23,W,Yankton,SD
46,35,59,N,120,30,36,W,Yakima,WA
42,16,12,N,71,48,0,W,Worcester,MA
43,37,48,N,89,46,11,W,WisconsinDells,WI
36,5,59,N,80,15,0,W,Winston-Salem,NC
49,52,48,N,97,9,0,W,Winnipeg,MB
39,11,23,N,78,9,36,W,Winchester,VA
34,14,24,N,77,55,11,W,Wilmington,NC
39,45,0,N,75,33,0,W,Wilmington,DE
48,9,0,N,103,37,12,W,Williston,ND
41,15,0,N,77,0,0,W,Williamsport,PA
37,40,48,N,82,16,47,W,Williamson,WV
33,54,0,N,98,29,23,W,WichitaFalls,TX
37,41,23,N,97,20,23,W,Wichita,KS
40,4,11,N,80,43,12,W,Wheeling,WV
26,43,11,N,80,3,0,W,WestPalmBeach,FL
47,25,11,N,120,19,11,W,Wenatchee,WA
41,25,11,N,122,23,23,W,Weed,CA
The first issue is here:
|> Stream.filter(fn
["LatD" | _] -> false
end)
all the lines should pass this and the only first one matches the given clauses. This would fix the issue
|> Stream.filter(fn
["LatD" | _] -> false
_ -> true
end)
or
|> Stream.reject(&match?(["LatD" | _], &1))
Enum.find(fn State -> String after looks unclear and would be surely the next issue. I failed to understand what have you tried to achieve here.
The general advice would be: don’t reinvent the wheel and use NimbleCSV written by José Valim to parse CSVs, because there are lot of corner cases (like commas inside quotes in any field etc,) handled properly in the aforementioned library.
Aleksei Matiushkin gave you the right answer but also you have this function:
fn
State ->
String
[LatD, LatM, LatS, NS, LonD, LonM, LonS, EW, City, State] ->
IO.puts("find -> #{State}")
true
end
It accepts two possible values, either State which is an atom, or a list of 10 specific atoms.
What you want to do is use variables, and variables in Elixir start with a lowercase letter or an underscore if it has to be ignored.
fn
state ->
String
[latd, latm, lats, ns, lond, lonm, lons, ew, city, state] ->
IO.puts("find -> #{state}")
true
end
But in this case, the first clause of the function will always match anything because it acts like a catch-all clause.
What you probably want is:
fn
[_latd, _latm, _lats, _ns, _lond, _lonm, _lons, _ew, _city, state] ->
IO.puts("find -> #{state}")
# here decide if you want to return true or false,
# for instance `state == NC`
true
end
I have a route in Phoenix that needs to check the JSON parameters with REGEX.
In my routine, I am creating a list of errors to report in the body in case one or more regexes fail.
But whenever I run the code, I get the "FunctionClauseError" error on "Regex.match?". I have tried String.match, but they evaluate to the same function.
Here is my code:
def postServidor(conn, parameters) do
reasons = []
error = False
if not Regex.match?(~r/^(19[0-9]{2}|2[0-9]{3})-(0[1-9]|1[012])-([123]0|[012][1-9]|31)$/, Map.get(parameters, "data_nascimento")) do
{error, reasons} = {True, reasons ++ [%{Reason => "[data_nascimento] missing or failed to match API requirements. It should look like this: 1969-02-12"}]}
end
if not Regex.match?(~r/^([A-Z][a-z]+([ ]?[a-z]?['-]?[A-Z][a-z]+)*)$/, Map.get(parameters, "nome")) do
{error, reasons} = {True, reasons ++ [%{Reason => "[name] missing or failed to match API requirements. It should look like this: Firstname Middlename(optional) Lastname"}]}
end
if not Regex.match?(~r/^([A-Z][a-z]+([ ]?[a-z]?['-]?[A-Z][a-z]+)*)$/, Map.get(parameters, "nome_identificacao")) do
{error, reasons} = {True, reasons ++ [%{Reason => "[nome_identificacao] missing or failed to match API requirements. It should look like this: Firstname Middlename(optional) Lastname"}]}
end
if not Regex.match?(~r/\b[MF]{1}\b/, Map.get(parameters, "sexo")) do
{error, reasons} = {True, reasons ++ [%{Reason => "[sexo] missing or failed to match API requirements. It should look like this: M for male, F for female"}]}
end
if not Regex.match?( ~r/\b[0-9]+\b/, Map.get(parameters, "id_pessoa")) do
{error, reasons} = {True, reasons ++ [%{Reason => "[id_pessoa] missing or failed to match API requirements. It should be numeric. "}]}
end
if not Regex.match?(~r/\b[0-9]+\b/, Map.get(parameters, "matricula_interna")) do
{error, reasons} = {True, reasons ++ [%{Reason => "[matricula_interna] missing or failed to match API requirements. It should be numeric. "}]}
end
if not Regex.match?(~r/\b[0-9]+\b/, Map.get(parameters, "siape")) do
{error, reasons} = {True, reasons ++ [%{Reason => "[siape] missing or failed to match API requirements. It should be numeric. "}]}
end
if error = True do
json put_status(conn, 400),reasons
else
IO.puts("ok")
end
end
Regex.match?/2 requires a string as its second argument. You're using Map.get/3 to potentially return that string, but if the key isn't found in the map, it defaults to nil, and that's my guess as to what's happening. If the key is not found in your map, it's passing nil to Regex.match?/2, for which there will be no function clause that matches. You can either fix your map to correctly have the key, or you can provide your own default string to use as a third parameter to Map.get/3. For example, Map.get(parameters, "data_nascimento", "some default").
In book "Programming Elixir" Dave Thomas shows an example
handle_open = fn
{:ok, file} -> "Read data: #{IO.read(file, :line)}"
{_, error} -> "Error: #{:file.format_error(error)}"
end
handle_open.(File.open("code/intro/hello.exs"))
handle_open.(File.open("nonexistent"))
I can't understand why second call of function goes into second flow.
Also why we don't pass an argument into func:
handle_open = fn(file)
?
This is a multi-clause anonymous function. handle_open has 2 clauses, one to match arguments with the pattern {:ok, file} and one with {_, error}. The first pattern that matches the arguments is executed. The given code is almost [1] equivalent to the following:
handle_open = fn arg ->
case arg do
{:ok, file} -> ...
{_, error} -> ...
end
end
So if the file exists, File.open will return {:ok, file} and the first clause will be executed. If it doesn't, {:error, error} will be returned which will match the second clause and execute that.
[1]: "Almost" because the error raised when none of the patterns match will be slightly different for the two cases.
This actually just mimics the pattern-match behaviour of Elixir.
Like you can have pattern match on your Module functions as well.
defmodule MyModule do
def read_file(path) do
File.open(path) |> handle_file()
end
defp handle_file({:ok, file}) do
// File there..
end
defp handle_file({_, error}) do
// Could not open file.. because of `error`
end
end
Like in the book example, I would try to open the file inside a case clause and delegate the outcome to the appropriate function:
defmodule FileReader do
def read_file(path) do
case File.open(path) do
{:ok, file} -> handle_file(file)
{_, error} -> handle_error(error)
end
end
defp handle_file(file) do
// File there..
end
defp handle_error(error) do
//
end
end
Doing upsert is common in my app and I want to implement the cleanest and simple way to implement upsert.
Should I use fragments to implement native sql upsert?
Any idiomatic ecto way to do upsert?
You can use Ecto.Repo.insert_or_update/2, please note that for this to work, you will have to load existing models from the database.
model = %Post{id: 'existing_id', ...}
MyRepo.insert_or_update changeset
# => {:error, "id already exists"}
Example:
result =
case MyRepo.get(Post, id) do
nil -> %Post{id: id} # Post not found, we build one
post -> post # Post exists, using it
end
|> Post.changeset(changes)
|> MyRepo.insert_or_update
case result do
{:ok, model} -> # Inserted or updated with success
{:error, changeset} -> # Something went wrong
end
In my case insert_or_update raised an error due to the unique index constraint 🤔
What did work for me was Postgres v9.5 upsert through on_conflict parameter:
(considering unique column is called user_id)
changeset
|> MyRepo.insert(
on_conflict: :replace_all,
conflict_target: :user_id
)
If you're looking to upsert by something other than id, you can swap in get_by for get like this:
model = %User{email: "existing_or_new_email#heisenberg.net", name: "Cat", ...}
model |> User.upsert_by(:email)
# => {:found, %User{...}} || {:ok, %User{...}}
defmodule App.User do
alias App.{Repo, User}
def upsert_by(%User{} = record_struct, selector) do
case User |> Repo.get_by({selector, record_struct |> Map.get(selector)}) do
nil -> %User{} # build new user struct
user -> user # pass through existing user struct
end
|> User.changeset(record_struct |> Map.from_struct)
|> Repo.insert_or_update
end
end
On the off chance you're looking for a flexible approach that works across models and for multiple selectors (ie country + passport number), check out my hex package EctoConditionals!