Related
I've been learning elixir this month, and was in a situation where I wanted to convert a binary object into a list of bits, for pattern matching.
My research led me here, to an article showing a method for doing so. However, I don't fully understand one of the arguments passed to the extract function.
I could just copy and paste the code, but I'd like to understand what's going on under the hood here.
The argument is this: <<b :: size(1), bits :: bitstring>>.
What I understand
I understand that << x >> denotes a binary object x. Logically to me, it looks as though this is similar to performing: [head | tail] = list on a List, to get the first element, and then the remaining ones as a new list called tail.
What I don't understand
However, I'm not familiar with the syntax, and I have never seen :: in elixir, nor have I ever seen a binary object separated by a comma: ,. I also, haven't seen size(x) used in Elixir, and have never encountered a bitstring.
The Bottom Line
If someone, could explain exactly how the syntax for this argument breaks down, or point me towards a resource I would highly appreciate it.
For your convenience, the code from that article:
defmodule Bits do
# this is the public api which allows you to pass any binary representation
def extract(str) when is_binary(str) do
extract(str, [])
end
# this function does the heavy lifting by matching the input binary to
# a single bit and sends the rest of the bits recursively back to itself
defp extract(<<b :: size(1), bits :: bitstring>>, acc) when is_bitstring(bits) do
extract(bits, [b | acc])
end
# this is the terminal condition when we don't have anything more to extract
defp extract(<<>>, acc), do: acc |> Enum.reverse
end
IO.inspect Bits.extract("!!") # => [0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1]
IO.inspect Bits.extract(<< 99 >>) #=> [0, 1, 1, 0, 0, 0, 1, 1]
Elixir pattern matching seems mind blowingly easy to use for
structured binary data.
Yep. You can thank the erlang inventors.
According to the documentation, <<x :: size(y)>> denotes a bitstring,
whos decimal value is x and is represented by a string of bits that is
y in length.
Let's dumb it down a bit: <<x :: size(y)>> is the integer x inserted into y bits. Examples:
<<1 :: size(1)>> => 1
<<1 :: size(2)>> => 01
<<1 :: size(3)>> => 001
<<2 :: size(3)>> => 010
<<2 :: size(4)>> => 0010
The number of bits in the binary type is divisible by 8, so a binary type has a whole number of bytes (1 byte = 8 bits). The number of bits in a bitstring is not divisible by 8. That's the difference between the binary type and the bitstring type.
I understand that << x >> denotes a binary object x. Logically to me,
it looks as though this is similar to performing: [head | tail] = list
on a List, to get the first element, and then the remaining ones as a
new list called tail.
Yes:
defmodule A do
def show_list([]), do: :ok
def show_list([head|tail]) do
IO.puts head
show_list(tail)
end
def show_binary(<<>>), do: :ok
def show_binary(<<char::binary-size(1), rest::binary>>) do
IO.puts char
show_binary(rest)
end
end
In iex:
iex(6)> A.show_list(["a", "b", "c"])
a
b
c
:ok
iex(7)> "abc" = <<"abc">> = <<"a", "b", "c">> = <<97, 98, 99>>
"abc"
iex(9)> A.show_binary(<<97, 98, 99>>)
a
b
c
:ok
Or you can interpret the integers in the binary as plain old integers:
def show(<<>>), do: :ok
def show(<<ascii_code::integer-size(8), rest::binary>>) do
IO.puts ascii_code
show(rest)
end
In iex:
iex(6)> A.show(<<97, 98, 99>>)
97
98
99
:ok
The utf8 type is super useful because it will grab as many bytes as required to get a whole utf8 character:
def show(<<>>), do: :ok
def show(<<char::utf8, rest::binary>>) do
IO.puts char
show(rest)
end
In iex:
iex(8)> A.show("ۑ")
8364
235
:ok
As you can see, the uft8 type returns the unicode codepoint of the character. To get the character as a string/binary:
def show(<<>>), do: :ok
def show(<<codepoint::utf8, rest::binary>>) do
IO.puts <<codepoint::utf8>>
show(rest)
end
You take the codepoint(an integer) and use it to create the binary/string <<codepoint::utf8>>.
In iex:
iex(1)> A.show("ۑ")
€
ë
:ok
You can't specify a size for the utf8 type, though, so if you want to read multiple utf8 characters, you have to specify multiple segments.
And of course, the segment rest::binary, i.e. a binary type with no size specified, is super useful. It can only appear at the end of a pattern, and rest::binary is like the greedy regex: (.*). The same goes for rest::bitstring.
Although the elixir docs don't mention it anywhere, the total number of bits in a segment, where a segment is one of those things:
| | |
v v v
<< 1::size(8), 1::size(16), 1::size(1) >>
is actually unit * size, where each type has a default unit. The default type for a segment is integer, so the type for each segment above defaults to integer. An integer has a default unit of 1 bit, so the total number of bits in the first segment is: 8 * 1 bit = 8 bits. The default unit for the binary type is 8 bits, so a segment like:
<< char::binary-size(6)>>
has a total size of 6 * 8 bits = 48 bits. Equivalently, size(6) is just the number of bytes. You can specify the unit just like you can the size, e.g. <<1::integer-size(2)-unit(3)>>. The total bit size of that segment is: 2 * 3 bits = 6 bits.
However, I'm not familiar with the syntax
Check this out:
def bitstr2bits(bitstr) do
for <<bit::integer-size(1) <- bitstr>>, do: bit
end
In iex:
iex(17)> A.bitstr2bits <<1::integer-size(2), 2::integer-size(2)>>
[0, 1, 1, 0]
Equivalently:
iex(3)> A.bitstr2bits(<<0b01::integer-size(2), 0b10::integer-size(2)>>)
[0, 1, 1, 0]
Elixir tends to abstract away recursion with library functions, so usually you don't have to come up with your own recursive definitions like at your link. However, that link shows one of the standard, basic recursion tricks: adding an accumulator to the function call to gather results that you want the function to return. That function could also be written like this:
def bitstr2bits(<<>>), do: []
def bitstr2bits(<<bit::integer-size(1), rest::bitstring>>) do
[bit | bitstr2bits(rest)]
end
The accumulator function at the link is tail recursive, which means it takes up a constant (small) amount of memory--no matter how many recursive function calls are needed to step through the bitstring. A bitstring with 10 million bits? Requiring 10 million recursive function calls? That would only require a small amount of memory. In the old days, the alternate definition I posted could potentially crash your program because it would take up more and more memory for each recursive function call, and if the bitstring were long enough the amount of memory needed would be too large, and you would get stackoverflow and your program would crash. However, erlang has optimized away the disadvantages of recursive functions that are not tail recursive.
You can read about all these here, short answer:
:: is similar as guard, like a when is_integer(a), in you case size(1) expect a 1 bit binary
, is a separator between matching binaries, like | in [x | []] or like comma in [a, b]
bitstring is a superset over binaries, you can read about it here and here, any binary can be respresented as bitstring
iex> ?h
104
iex> ?e
101
iex> ?l
108
iex> ?o
111
iex> <<104, 101, 108, 108, 111>>
"hello"
iex> [104, 101, 108, 108, 111]
'hello'
but not vice versa
iex> <<1, 2, 3>>
<<1, 2, 3>>
After some research, I realized I overlooked some important information located at: elixir-lang.
According to the documentation, <<x :: size(y)>> denotes a bitstring, whos decimal value is x and is represented by a string of bits that is y in length.
Furthermore, <<binary>> will always attempt to conglomerate values in a left-first direction, into bytes or 8-bits, however, if the number of bits is not divisible by 8, there will by x bytes, followed by a bitstring.
For example:
iex> <<3::size(5), 5::size(6)>> # <<00011, 000101>>
<<24, 5::size(3)>> # automatically shifted to: <<00011000(24) , 101>>
Now, elixir also lets us pattern match binaries, and bitstrings like so:
iex> <<3 :: size(2), b :: bitstring>> = <<61 :: size(6)>> # [11] [1101]
iex> b
<<13 :: size(4)>> # [1101]
So, i completly misunderstood binaries and biststrings, and pattern matching between the two.
Not really the answer to the question stated, but I’d put it here for the sake of formatting. In elixir we usually use Kernel.SpecialForms.for/1 comprehension for bitstring generation.
for << b :: size(1) <- "!!" >>, do: b
#⇒ [0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1]
for << b :: size(1) <- <<99>> >>, do: b
#⇒ [0, 1, 1, 0, 0, 0, 1, 1]
I wanted to use the bits, in an 8 bit binary to toggle conditions. So
[b1, b2, ...] = extract(<<binary>>)
I then wanted to say:
if b1, do: x....
if b2, do: y...
Is there a better way to do what I'm trying to do, instead of pattern
matching?
First of all, the only terms that evaluate to false in elixir are false and nil (just like in ruby):
iex(18)> x = 1
1
iex(19)> y = 0
0
iex(20)> if x, do: IO.puts "I'm true."
I'm true.
:ok
iex(21)> if y, do: IO.puts "I'm true."
I'm true.
:ok
Although, the fix is easy:
if b1 == 1, do: ...
Extracting the bits into a list is unnecessary because you can just iterate the bitstring:
def check_bits(<<>>), do: :ok
def check_bits(<<bit::integer-size(1), rest::bitstring>>) do
if bit == 1, do: IO.puts "bit is on"
check_bits(rest)
end
In other words, you can treat a bitstring similarly to a list.
Or, instead of performing the logic in the body of the function to determine whether the bit is 1, you can use pattern matching in the head of the function:
def check_bits(<<>>), do: :ok
def check_bits(<< 1::integer-size(1), rest::bitstring >>) do
IO.puts "The bit is 1."
check_bits(rest)
end
def check_bits(<< 0::integer-size(1), rest::bitstring >>) do
IO.puts "The bit is 0."
check_bits(rest)
end
Instead of using a variable, bit, for the match like here:
bit::integer-size(1)
...you use a literal value, 1:
1::integer-size(1)
The only thing that can match a literal value is the literal value itself. As a result, the pattern:
<< 1::integer-size(1), rest::bitstring >>
will only match a bitstring where the first bit, integer-size(1), is 1. The literal matching employed there is similar to doing the following with a list:
def list_contains_4([4|_tail]) do
IO.puts "found a 4"
true #end the recursion and return true
end
def list_contains_4([head|tail]) do
IO.puts "#{head} is not a 4"
list_contains_4(tail)
end
def list_contains_4([]), do: false
The first function clause tries to match the literal 4 at the head of the list. If the head of the list is not 4, there's no match; so elixir moves on to the next function clause, and in the next function clause the variable head will match anything in the list.
Using pattern matching in the head of a function rather than performing logic in the body of a function is considered more stylish and efficient in erlang.
I’m writing Julia code whose inputs are json files, that performs analysis in (the field of mathematical finance) and writes results as json. The code is a port from R in the hope of performance improvement.
I parse the input files using JSON.parsefile. This returns a Dict in which I observe that all vectors are of type Array{Any,1}. As it happens, I know that the input file will never contain vectors of mixed type, such as some Strings and some Numbers.
So I wrote the following code, which seems to work well and is “safe” in the sense that if the calls to convert fail then a vector continues to have type Array{Any,1}.
function typenarrow!(d::Dict)
for k in keys(d)
if d[k] isa Array{Any,1}
d[k] = typenarrow(d[k])
elseif d[k] isa Dict
typenarrow!(d[k])
end
end
end
function typenarrow(v::Array{Any,1})
for T in [String,Int64,Float64,Bool,Vector{Float64}]
try
return(convert(Vector{T},v))
catch; end
end
return(v)
end
My question is: Is this worth doing? Can I expect code that processes the contents of the Dict to execute faster if I do this type narrowing? I think the answer is yes in that the Julia performance tips recommend to “Annotate values taken from untyped locations” and this approach ensures there are no “untyped locations”.
There are two levels of the answer to this question:
Level 1
Yes, it will help the performance of the code. See for instance the following benchmark:
julia> using BenchmarkTools
julia> x = Any[1 for i in 1:10^6];
julia> y = [1 for i in 1:10^6];
julia> #btime sum($x)
26.507 ms (477759 allocations: 7.29 MiB)
1000000
julia> #btime sum($y)
226.184 μs (0 allocations: 0 bytes)
1000000
You can write your typenarrow function using a bit simpler approach like this:
typenarrow(x) = [v for v in x]
as using the comprehension will produce a vector of concrete type (assuming your source vector is homogeneous)
Level 2
This is not fully optimal. The problem that is still left is that you have a Dict that is a container with abstract type parameter (see https://docs.julialang.org/en/latest/manual/performance-tips/#Avoid-containers-with-abstract-type-parameters-1). Therefore in order for the computations to be fast you have to use a barrier function (see https://docs.julialang.org/en/latest/manual/performance-tips/#kernel-functions-1) or use type annotation for variables you introduce (see https://docs.julialang.org/en/v1/manual/types/index.html#Type-Declarations-1).
In the ideal world your Dict would have keys and values of homogeneous types and all would be maximally fast then, but if I understand your code correctly values in your case are not homogeneous.
EDIT
In order to solve the Level 2 isuue you can convert Dict into NamedTuple like this (this is a minimal example assuming that Dicts only nest in Dicts directly, but it should be easy enough to extend if you want more flexibility).
First, the function performing the conversion looks like:
function typenarrow!(d::Dict)
for k in keys(d)
if d[k] isa Array{Any,1}
d[k] = [v for v in d[k]]
elseif d[k] isa Dict
d[k] = typenarrow!(d[k])
end
end
NamedTuple{Tuple(Symbol.(keys(d)))}(values(d))
end
Now a MWE of its use:
julia> using JSON
julia> x = """
{
"name": "John",
"age": 27,
"values": {
"v1": [1,2,3],
"v2": [1.5,2.5,3.5]
},
"v3": [1,2,3]
}
""";
julia> j1 = JSON.parse(x)
Dict{String,Any} with 4 entries:
"name" => "John"
"values" => Dict{String,Any}("v2"=>Any[1.5, 2.5, 3.5],"v1"=>Any[1, 2, 3])
"age" => 27
"v3" => Any[1, 2, 3]
julia> j2 = typenarrow!(j1)
(name = "John", values = (v2 = [1.5, 2.5, 3.5], v1 = [1, 2, 3]), age = 27, v3 = [1, 2, 3])
julia> dump(j2)
NamedTuple{(:name, :values, :age, :v3),Tuple{String,NamedTuple{(:v2, :v1),Tuple{Array{Float64,1},Array{Int64,1}}},Int64,Array{Int64,1}}}
name: String "John"
values: NamedTuple{(:v2, :v1),Tuple{Array{Float64,1},Array{Int64,1}}}
v2: Array{Float64}((3,)) [1.5, 2.5, 3.5]
v1: Array{Int64}((3,)) [1, 2, 3]
age: Int64 27
v3: Array{Int64}((3,)) [1, 2, 3]
The beauty of this approach is that Julia will know all types in j2, so if you pass j2 to any function as a parameter all calculations inside this function will be fast.
The downside of this approach is that a function taking j2 has to be pre-compiled, which might be problematic if j2 structure is huge (as then the structure of resulting NamedTuple is complex) and the amount of work your function does is relatively small. But for small JSON-s (small in the sense of structure, as vectors held in them can be large - their size does not add to the complexity) this approach has proven to be efficient in several applications I have developed.
I'm trying to wrap my head around how the return call works in a function. In the example below, I'm assigning 5 to number1 and 6 to number2. Then I return both below. When I print the output, I only get "5" as a result.
Can someone please explain why it's doing this? Why does it not print both numbers?
Thanks!
def numberoutput ():
number1 = 5
number2 = 6
return number1
return number2
print (numberoutput())
Here's a compact way to do the loops that you ask for. Your lists should not contain 1, though.
>>> list1 = list(range(2,11))
>>> list2 = list(range(2,11))
>>> primes = [a for a in list1 if all((a % b) != 0 for b in list2 if a != b) ]
>>> primes
[2, 3, 5, 7]
There are no duplicates in the results, because the comprehension just collects elements of list1. But there are plenty of ways to improve prime number detection, of course. This just shows you how to apply comprehensions to your algorithm.
Try this (change 10 by the number you want)
primes = []
for number in range(1,10):
is_prime = True
for div in range(2, number-1):
if number % div == 0:
is_prime = False
break
if is_prime:
primes.append(number)
Be careful though, this is not efficient at all. A little improvment is to change (number - 1) by int(sqrt(number)). But that's math rules. If you want the first 1000000 primes, that won't work. You wanna perhaps check more advanced methods to find primes if you need more.
Explanation:
you iterate first with all numbers between 1 and 10 - 1 = 9. This number is store into the variable "number". Then you iterate other the possible dividers. If the modulo for each pair of number and divider is 0, then it is not a prime number, you can mark it as not prime (is_prime = False) then quit your loop. At the end of the inner loop, you check the boolean is_prime and then add to the list if the boolean is set at True.
Here's a reasonably efficient way to find primes with a list comprehension, although it's not as efficient as the code by Robert William Hanks that I linked in the comments.
We treat 2 as a special case so we don't need to bother with any higher even numbers. And we only need to check factors less than the square root of the number we're testing.
from math import floor, sqrt
primes = [2] + [i for i in range(3, 100, 2)
if all(i % j != 0 for j in range(3, 1 + floor(sqrt(i)), 2))]
print(primes)
output
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]
Here's an alternative (less efficient) version that iterates over your list1 (which isn't really a list, it's a range object).
list1 = range(2, 100)
primes = [i for i in list1 if not [j for j in list1 if j*j <= i and i % j == 0]]
print(primes)
def call(nums):
nums[:] = [x for x in nums if x != 4]
numbers = [4, 5]
print(numbers)
call(numbers)
print(numbers)
The output for the above code is:
[4, 5]
[5]
But if you remove the "[:]", the output becomes the following:
[4, 5]
[4, 5]
I know that [:] makes a copy of the full list, but why the function argument is modified in one case and not in the other?
As you suspected the issue is hiding in the slicer in line:
nums[:] = [x for x in nums if x != 4]
Python is "pass by value" which means that you're not passing the pointer numbers into the function, but a copy of it. So re-assigning a value to a copy of the reference will not change the original reference.
But, when you use the slicer, you're accessing the object behind the reference and changing it directly which is why you see the side effect after you exit the function - when using the "slice".
As Python passes objects by reference, when you execute your function with nums[:] = ... line, you change actual list that you've passed from outside, numbers. When you change line to nums = ..., you just overwrite local variable called nums, without affecting numbers array.
In Python, you can not only slice collections to read them, but you can assign to slices, replacing sliced content.
For example:
>>> a = [0, 1, 2, 3, 4, 5]
>>> a[1:4]
[1, 2, 3]
If you assign to slice, it will replace part of original array:
>>> a[1:4] = ['z']
>>> a
[0, 'z', 4, 5]
But when assigning to slices, array remains the same:
>>> a = [0, 1, 2, 3, 4, 5]
>>> b = a
>>> a[:] = ['z']
>>> a
['z']
>>> b
['z']
As you can see, a and b change at the same time, because when assigning to slice, you don't change object's identity, you only change its contents.
>>> a = [0, 1, 2, 3, 4, 5]
>>> b = a
>>> a = ['z']
>>> a
['z']
>>> b
[0, 1, 2, 3, 4, 5]
This is not the case when you just assign to variable, dropping older array out of scope.
def call(nums):
nums = [x for x in nums if x != 4]
Would only change the value of the name nums function parameter which would accomplish nothing.
def call(nums):
nums[:] = [x for x in nums if x != 4]
Changes the actual value of the list passed in as an argument.
I'm working my way through Dave's upcoming book on Elixir, and in one exercise I would like to dynamically construct a function reference to Kernel.+/2, Kernel.-/2 etc, based on the contents of one character of a string, '+', '-' and so on.
Based on another SO question I expected to be able to call apply/3 passing Kernel, :+ and the two numbers like this:
apply(Kernel, :+, [5, 7])
This doesn't work because (if I understand right) Kernel.+/2 is a macro, not a function. I looked up the source code, and + is defined in terms of __op__, and I can call it from iex:
__op__(:+, 5, 7)
This works until I put the :+ into a variable:
iex(17)> h = list_to_atom('+')
:+
iex(18)> __op__(h, 5, 7)
** (CompileError) iex:18: undefined function __op__/3
src/elixir.erl:151: :elixir.quoted_to_erl/3
src/elixir.erl:134: :elixir.eval_forms/4
And I'm guessing there's no way to call __op__ using apply/3.
Of course, the brute-force method gets the job done.
defp _fn(?+), do: &Kernel.+/2
defp _fn(?-), do: &Kernel.-/2
defp _fn(?*), do: &Kernel.*/2
# defp _fn(?/), do: &Kernel.//2 # Nope, guess again
defp _fn(?/), do: &div/2 # or &(&1 / &2) or ("#{div &1, &2} remainder #{rem &1, &2}")
But is there something more concise and dynamic?
José Valim nailed it with his answer below. Here's the code in context:
def calculate(str) do
{x, op, y} = _parse(str, {0, :op, 0})
apply :erlang, list_to_atom(op), [x, y]
end
defp _parse([] , acc ) , do: acc
defp _parse([h | t], {a, b, c}) when h in ?0..?9, do: _parse(t, {a, b, c * 10 + h - ?0})
defp _parse([h | t], {_, _, c}) when h in '+-*/', do: _parse(t, {c, [h], 0})
defp _parse([_ | t], acc ) , do: _parse(t, acc)
You can just use the Erlang one:
apply :erlang, :+, [1,2]
We are aware this is confusing and we are studying ways to make it or more explicit or more transparent.
UPDATE: Since Elixir 1.0, you can dispatch directly to Kernel (apply Kernel, :+, [1, 2]) or even use the syntax the OP first attempted (&Kernel.+/2).