I have a html page that contains a navigation bar at the top of the screen. In the navigation bar, I have a search box and what I want it to do is, you type in this box, hit enter and the results are displayed as a dropdown menu
<li><input type="text" id="search-bar" placeholder="Search"></li>
This is the html search input box. I have given it an id search-bar to eventually create the dropdown menu in ClojureScript
(when-let [section (. js/document (getElementById "search-bar"))]
(r/render-component [search-bar-component] section))
Currently I have a search-form that looks like the following
(defn search-form
[]
[:div
[:p "What are you searching for? "
[:input
{:type :text
:name :search
:on-change #(do
(swap! fields assoc :search (-> % .-target .-value))
(search-index (:search #fields)))
:value (:search #fields)}]]
[:p (create-links #search-results)]])
(defn- search-component
[]
[search-form])
This is my search-component.
What I want to happen is when you type in the input box on the navbar (say "hello", it calls search-index from the search-form with the parameter being the value you type in ("hello") and then returns the results as a dropdown menu below.
search-form works right now as a form on a html page, where you input some text into a form and then the results are displayed below. I want to change it to be on the navbar instead of as a separate page, where the input form is on the navbar and the results are displayed below
How would I have to change my search-form in order to do this?
I think I can do something along the lines of this
(defn search-bar-form
[]
[:div
[:input
{:type :text
:name :search
:on-change #(do
(swap! fields assoc :search (-> % .-target .-value))
(search-index (:search #fields)))
:value (:search #fields)}]
[:p (create-links #search-results)]])
(defn- search-bar-component
[]
[search-form])
Any help would be much appreciated.
re-com provides a typeahead component. It looks like you're using reagent, so you could just use that, otherwise you can use it as inspiration.
Related
I'm scraping a website with some poorly structured HTML using a Clojure wrapper around jsoup called Reaver. Here is an example of some of the HTML structure:
<div id="article">
<aside>unwanted text</aside>
<p>Some text</p>
<nav><ol><li><h2>unwanted text</h2></li></ol></nav>
<p>More text</p>
<h2>A headline</h2>
<figure><figcaption>unwanted text</figcaption></figure>
<p>More text</p>
Here is a paragraph made of some raw text directly in the div
<p>Another paragraph of text</p>
More raw text and this one has an <a>anchor tag</a> inside
<dl>
<dd>unwanted text</dd>
</dl>
<p>Etc etc</p>
</div>
This div represents an article on a wiki. I want to extract the text from it, but as you can see, some paragraphs are in p tags, and some are contained directly within the div. I also need the headlines and anchor tag text.
I know how to parse and extract the text from all of the p, a, and h tags, and I can select for the div and extract the inner text from it, but the problem is that I end up with two selections of text that I need to merge somehow.
How can I extract the text from this div, so that all of the text from the p, a, h tags, as well as the inner text on the div, are extracted in order? The result should be paragraphs of text in the same order as what is in the HTML.
Here is what I am currently using to extract, but the inner div text is missing from the results:
(defn get-texts [url]
(:paragraphs (extract (parse (slurp url))
[:paragraphs]
"#article > *:not(aside, nav, table, figure, dl)" text)))
Note also that additional unwanted elements appear in this div, e.g., aside, figure, etc. These elements contain text, as well as nested elements with text, that should not be included in the result.
You could extract the entire article as a JSoup object (likely an Element), then convert it to an EDN representation using reaver/to-edn. Then you go through the :content of that and handle both strings (the result of TextNodes) and elements that have a :tag that interests you.
(Code by vaer-k)
(defn get-article [url]
(:article (extract (parse (slurp url))
[:article]
"#article"
edn)))
(defn text-elem?
[element]
(or (string? element)
(contains? #{:p :a :b :i} (:tag element))))
(defn extract-text
[{content :content}]
(let [text-children (filter text-elem? content)]
(reduce #(if (string? %2)
(str %1 %2)
(str %1 (extract-text %2)))
""
text-children)))
(defn extract-article [url]
(-> url
get-article
extract-text))
You can solve this using the tupelo.forest library, which was presented in an "Unsession" of the Clojure/Conj 2019 just last week.
Below is the solution written as a unit test. First some declarations and the sample data:
(ns tst.demo.core
(:use tupelo.forest tupelo.core tupelo.test)
(:require
[clojure.string :as str]
[schema.core :as s]
[tupelo.string :as ts]))
(def html-src
"<div id=\"article\">
<aside>unwanted text</aside>
<p>Some text</p>
<nav><ol><li><h2>unwanted text</h2></li></ol></nav>
<p>More text</p>
<h2>A headline</h2>
<figure><figcaption>unwanted text</figcaption></figure>
<p>More text</p>
Here is a paragraph made of some raw text directly in the div
<p>Another paragraph of text</p>
More raw text and this one has an <a>anchor tag</a> inside
<dl>
<dd>unwanted text</dd>
</dl>
<p>Etc etc</p>
</div> ")
To start off, we add the html data (a tree) to the forest after removing all newlines, etc. This uses the Java "TagSoup" parser internally:
(dotest
(hid-count-reset)
(with-forest (new-forest)
(let [root-hid (add-tree-html
(ts/collapse-whitespace html-src))
unwanted-node-paths (find-paths-with root-hid [:** :*]
(s/fn [path :- [HID]]
(let [hid (last path)
node (hid->node hid)
tag (grab :tag node)]
(or
(= tag :aside)
(= tag :nav)
(= tag :figure)
(= tag :dl)))))]
(newline) (spyx-pretty :html-orig (hid->bush root-hid))
The spyx-pretty shows the "bush" format of the data (similar to hiccup format):
:html-orig (hid->bush root-hid) =>
[{:tag :html}
[{:tag :body}
[{:id "article", :tag :div}
[{:tag :aside, :value "unwanted text"}]
[{:tag :p, :value "Some text"}]
[{:tag :nav}
[{:tag :ol} [{:tag :li} [{:tag :h2, :value "unwanted text"}]]]]
[{:tag :p, :value "More text"}]
[{:tag :h2, :value "A headline"}]
[{:tag :figure} [{:tag :figcaption, :value "unwanted text"}]]
[{:tag :p, :value "More text"}]
[{:tag :tupelo.forest/raw,
:value
" Here is a paragraph made of some raw text directly in the div "}]
[{:tag :p, :value "Another paragraph of text"}]
[{:tag :tupelo.forest/raw,
:value " More raw text and this one has an "}]
[{:tag :a, :value "anchor tag"}]
[{:tag :tupelo.forest/raw, :value " inside "}]
[{:tag :dl} [{:tag :dd, :value "unwanted text"}]]
[{:tag :p, :value "Etc etc"}]]]]
So we can see the data has been loaded correctly. Now, we want to remove all of the unwanted nodes as identified by the find-paths-with. Then, print the modified tree:
(doseq [path unwanted-node-paths]
(remove-path-subtree path))
(newline) (spyx-pretty :html-cleaned (hid->bush root-hid))
:html-cleaned (hid->bush root-hid) =>
[{:tag :html}
[{:tag :body}
[{:id "article", :tag :div}
[{:tag :p, :value "Some text"}]
[{:tag :p, :value "More text"}]
[{:tag :h2, :value "A headline"}]
[{:tag :p, :value "More text"}]
[{:tag :tupelo.forest/raw,
:value
" Here is a paragraph made of some raw text directly in the div "}]
[{:tag :p, :value "Another paragraph of text"}]
[{:tag :tupelo.forest/raw,
:value " More raw text and this one has an "}]
[{:tag :a, :value "anchor tag"}]
[{:tag :tupelo.forest/raw, :value " inside "}]
[{:tag :p, :value "Etc etc"}]]]]
At this point, we simply walk the tree and accumulate any surviving text nodes into a vector:
(let [txt-accum (atom [])]
(walk-tree root-hid
{:enter (fn [path]
(let [hid (last path)
node (hid->node hid)
value (:value node)] ; may not be present
(when (string? value)
(swap! txt-accum append value))))})
To verify, we compare the found text nodes (ignoring whitespace) to the desired result:
(is-nonblank= (str/join \space #txt-accum)
"Some text
More text
A headline
More text
Here is a paragraph made of some raw text directly in the div
Another paragraph of text
More raw text and this one has an
anchor tag
inside
Etc etc")))))
For more details, see the README file and the API docs. Be sure to also view the Lightning Talk for an overview.
I have a html page, with one structure that I want to turn into Clojure data structure. I’m hitting a mental block on how to approach this in an idiomatic way
This is the structure I have:
<div class=“group”>
<h2>title1<h2>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading1</h3>
<a href=“path1” />
</div>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading2</h3>
<a href=“path2” />
</div>
</div>
<div class=“group”>
<h2>title2<h2>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading3</h3>
<a href=“path3” />
</div>
</div>
Structure I want:
'(
[“Title1” “subhead1” “path1”]
[“Title1” “subhead2” “path2”]
[“Title2” “subhead3” “path3”]
[“Title3” “subhead4” “path4”]
[“Title3” “subhead5” “path5”]
[“Title3” “subhead6” “path6”]
)
The repetition of titles is intentional.
I’ve read David Nolan’s enlive tutorial. That offers a good solution if there was a parity between group and subgroup, but in this case it can be random.
Thanks for any advice.
You can use Hickory for parsing, and then Clojure has some very nice tools for transforming the parsed HTML to the form you want:
(require '[hickory.core :as html])
(defn classifier [tag klass]
(comp #{[:element tag klass]} (juxt :type :tag (comp :class :attrs))))
(def group? (classifier :div "“group”"))
(def subgroup? (classifier :div "“subgroup”"))
(def path? (classifier :a nil))
(defn identifier? [tag] (classifier tag nil))
(defn only [x]
;; https://stackoverflow.com/a/14792289/5044950
{:pre [(seq x)
(nil? (next x))]}
(first x))
(defn identifier [tag element]
(->> element :content (filter (identifier? tag)) only :content only))
(defn process [data]
(for [group (filter group? (map html/as-hickory (html/parse-fragment data)))
:let [title (identifier :h2 group)]
subgroup (filter subgroup? (:content group))
:let [subheading (identifier :h3 subgroup)]
path (filter path? (:content subgroup))]
[title subheading (:href (:attrs path))]))
Example:
(require '[clojure.pprint :as pprint])
(def data
"<div class=“group”>
<h2>title1</h2>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading1</h3>
<a href=“path1” />
</div>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading2</h3>
<a href=“path2” />
</div>
</div>
<div class=“group”>
<h2>title2</h2>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading3</h3>
<a href=“path3” />
</div>
</div>")
(pprint/pprint (process data))
;; (["title1" "subheading1" "“path1”"]
;; ["title1" "subheading2" "“path2”"]
;; ["title2" "subheading3" "“path3”"])
The solution can be splited in two parts
Parsing: parse it with clojure html parser or any other parser.
Custom data structure: modify the parsed html, you can use clojure.walk for that if you want.
You can solve this problem with the tupelo.forest library. Here is an annotated unit test showing the approach. You can find more information in the API docs and both the unit tests and the example demos. Additional documentation is forthcoming.
(dotest
(with-forest (new-forest)
(let [html-str "<div class=“group”>
<h2>title1</h2>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading1</h3>
<a href=“path1” />
</div>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading2</h3>
<a href=“path2” />
</div>
</div>
<div class=“group”>
<h2>title2</h2>
<div class=“subgroup”>
<p>unused</p>
<h3>subheading3</h3>
<a href=“path3” />
</div>
</div>"
enlive-tree (->> html-str
java.io.StringReader.
en-html/html-resource
first)
root-hid (add-tree-enlive enlive-tree)
tree-1 (hid->hiccup root-hid)
; Removing whitespace nodes is optional; just done to keep things neat
blank-leaf-hid? (fn fn-blank-leaf-hid? ; whitespace pred fn
[hid]
(let [node (hid->node hid)]
(and (contains-key? node ::tf/value)
(ts/whitespace? (grab ::tf/value node)))))
blank-leaf-hids (keep-if blank-leaf-hid? (all-leaf-hids)) ; find whitespace nodes
>> (apply remove-hid blank-leaf-hids) ; delete whitespace nodes found
tree-2 (hid->hiccup root-hid)
>> (is= tree-2 [:html
[:body
[:div {:class "“group”"}
[:h2 "title1"]
[:div {:class "“subgroup”"}
[:p "unused"]
[:h3 "subheading1"]
[:a {:href "“path1”"}]]
[:div {:class "“subgroup”"}
[:p "unused"]
[:h3 "subheading2"]
[:a {:href "“path2”"}]]]
[:div {:class "“group”"}
[:h2 "title2"]
[:div {:class "“subgroup”"}
[:p "unused"]
[:h3 "subheading3"]
[:a {:href "“path3”"}]]]]])
; find consectutive nested [:div :h2] pairs at any depth in the tree
div-h2-paths (find-paths root-hid [:** :div :h2])
>> (is= (format-paths div-h2-paths)
[[{:tag :html}
[{:tag :body}
[{:class "“group”", :tag :div}
[{:tag :h2, :tupelo.forest/value "title1"}]]]]
[{:tag :html}
[{:tag :body}
[{:class "“group”", :tag :div}
[{:tag :h2, :tupelo.forest/value "title2"}]]]]])
; find the hid for each top-level :div (i.e. "group"); the next-to-last (-2) hid in each vector
div-hids (mapv #(idx % -2) div-h2-paths)
; for each of div-hids, find and collect nested :h3 values
dif-h3-paths (vec
(lazy-gen
(doseq [div-hid div-hids]
(let [h2-value (find-leaf-value div-hid [:div :h2])
h3-paths (find-paths div-hid [:** :h3])
h3-values (it-> h3-paths (mapv last it) (mapv hid->value it))]
(doseq [h3-value h3-values]
(yield [h2-value h3-value]))))))
]
(is= dif-h3-paths
[["title1" "subheading1"]
["title1" "subheading2"]
["title2" "subheading3"]])
)))
I am new to Clojure and I am trying to make a page where you can see all the news that are in a table on the left, and only sports news on the right of the page. I tried to add a new parameter to the clostache/render:
(defn render-template [template-file params param]
(clostache/render (read-template template-file) params param))
(defn welcome []
(render-template "index" {:sports (model/justSports)} {:news (model/all)}))
where the model/all and model/justSports are:
(defn all []
(j/query mysql-db
(s/select * :news)))
(defn justSports []
(j/query mysql-db
(s/select * :news ["genre = ?" "sports"])))
and the news should be shown like this:
<div style="background-color: #D3D3D3; width: 450px; height: 800px; position: absolute; right: 10px; margin-top: 10px; border-radius: 25px;">
<sections>
{{#sports}}
<h2>{{title}}</h2>
<p>{{text}}<p>
{{/sports}}
</sections>
</div>
<div class="container" style="width: 500px; height: 800px; position: absolute; left: 20px;">
<h1>Listing Posts</h1>
<sections>
{{#news}}
<h2>{{title}}</h2>
<p>{{text}}<p>
{{/news}}
</sections>
</div>
But it doesn't work. It just shows the data from the first parameter on the page. What do you think, how can I make this work?
P.S.
Don't mind the ugly css, I will work on that :)
The following should make it work:
(defn render-template [template-file params]
(clostache/render (read-template template-file) params))
(defn welcome []
(render-template "index" {:sports (model/justSports)
:news (model/all)}))
render has three "arities":
(defn render
"Renders the template with the data and, if supplied, partials."
([template]
(render template {} {}))
([template data]
(render template data {}))
([template data partials]
(replace-all (render-template template data partials)
[["\\\\\\{\\\\\\{" "{{"]
["\\\\\\}\\\\\\}" "}}"]])))
You were calling the 3-arity overload which takes [template data partials], so the second map with the :news key was being taken as the partials by clostache. You want to call the 2-arity version which takes just [template data], passing one map with keys :news & :sports.
I am new to Watir and Cucumber, and I am trying to run an automation to create Live IDs. Below is the HTML for the link that I want to click, the "New" text is what it is showing on webpage. It would lead me to the form to add new contact to my Live account.
<ul class="c_cc" role="presentation" styple="overflow:visible;">
<li class="c_sm c_mcp" id = "new">
<a id href="#" class="c_nobdr t_prs">
<span class="is_c" dir="ltr" style="padding-right: 5px;">
<img class="is_img" src="https://p.pfx.ms/is/invis.gif" onload="this.onload=null;$Do.when('$IS.Init',0,this);" style="width:26px;height:26px;background-position:-1px -1px;background-image:url('https://p.pfx.ms/h/command4.png');" alt="New contact" title />
</span>
"New"
</a>
<span class="c_ms"></span>
</li>
</ul>
the watir code I wrote to click the "New" is below:
#browser.div(:id, "c_header").div(:id, "c_cb0").ul(:class, "c_cc").span(:text, "is_c").when_present.click
I get this error:
Watir::Wait::TimeoutError: timed out after 30 seconds, waiting for {:id=>"is_c", :tag_name=>"span"} to become present
Then I tried below code:
#browser.div(:id, "c_header").div(:id, "c_cb0").ul(:class, "c_cc").span(:text, "New").when_present.click
but this code does not really clicking on the "New" link, so the next form won't show up, and the rest of the code cannot run. Does anyone know any solution to this problem?
I found out a new window popped up, so it could not find the element in the old window. Thanks guys for helping.
In the first watir code snippet, a :text locator is being used for the .span method instead of a :class locator. For example:
browser.ul(:class, "c_cc").span(:text, "is_c").exists? #=> false
browser.ul(:class, "c_cc").span(:class, "is_c").exists? #=> true
In the second watir code snippet, a :text locator with a value of "New" is being used for the .span method. In this case, a .link method should be used. Additionally, the string includes double-quotes, so the double-quotes must be escaped if enclosed in another set of double-quotes (or enclose in single-quotes). For example:
browser.ul(:class, "c_cc").span(:text, "New").exists? #=> false
browser.ul(:class, "c_cc").span(:text, "\"New\"").exists? #=> false
browser.ul(:class, "c_cc").link(:text, "\"New\"").exists? #=> true
browser.ul(:class, "c_cc").link(:text, '"New"').exists? #=> true
So, one of the following examples should work:
browser.link(:text, "\"New\"").when_present.click
browser.link(:text, '"New"').when_present.click
browser.link(:class, "c_nobdr t_prs").when_present.click
Wow. This is old and I have encountered the same behavior. First, this is in a Cucumber step definition. The following works perfectly in open Ruby code. It simply refuses to function as a step.
It finds the link in the table and clicks it. Click doesn't function.
link = browser.table(class: 'alert-table').tbody.rows[1].cells.last.link(text: 'View')
# <Watir::Anchor: located: false; {:class=>"alert-table", :tag_name=>"table"} --> {:tag_name=>"tbody"} --> {:index=>1} --> {:index=>-1} --> {:text=>"View", :tag_name=>"a"}>
link.click
Ruby 2.4
watir (6.16.5)
regexp_parser (~> 1.2)
selenium-webdriver (~> 3.6)
I'm trying to generate a link using the link_to helper that will output the following HTML:
<i class="some_class"></i>Link Name
However the code I'm using to try to accomplish this:
link_to(tag("i", class: options[:icon]) + title, url)
...is outputting:
<i class="some_class">Link Name</i>
Why is it doing this, and how can I fix it? Thanks.
EDIT:
I believe I found the issue.
<i> tags are not self-closable tags in HTML5. Therefore the text after the i is treated as that element's content.
Have you tried using the block format of link_to?
<%= link_to url do %>
<%= tag("i", class: options[:icon]) %>
Link Name
<% end %>
Tweak that to your needs and maybe you'll get what you're looking for.
This is the icon tag helper I use in my applications which I frequently pass as the first argument to link_to, which can either be used to create a icon tag alone, or an icon tag followed by text.
def icon_tag(icon, *args)
options = args.extract_options!
text = args.first || options.delete(:text)
if text.nil?
content_tag :i, "", class: ["icon", "icon-#{icon}"] + options[:class].to_a
else
"#{icon_tag icon} #{text}".html_safe
end
end