I have a method to parse website with using Swiftsoup go get the price of a product:
#objc func actionButtonTapped(){
let url = "https://www.overkillshop.com/de/c2h4-interstellar-liaison-panelled-zip-up-windbreaker-r001-b012-vanward-black-grey.html"
let url2 = "https://www.asos.com/de/asos-design/asos-design-schwarzer-backpack-mit-ringdetail-und-kroko-muster/prd/14253083?clr=schwarz&colourWayId=16603012&SearchQuery=&cid=4877"
do {
let html: String = getHTMLfromURL(url: url2)
let doc: Document = try SwiftSoup.parse(html)
let priceClasses: Elements = try doc.select("[class~=(?i)price]")
for priceClass: Element in priceClasses.array() {
let priceText : String = try priceClass.text()
print(try priceClass.className())
print("pricetext: \(priceText)")
}
} catch Exception.Error(let type, let message) {
print(message)
} catch {
print("error")
}
}
The method works fine for url but for url2 it is not printing all all the classNames even though they match the regex. This is where the price actually is:
<span data-id="current-price" data-bind="text: priceText(), css: {'product-price-discounted' : isDiscountedPrice }, markAndMeasure: 'pdp:price_displayed'" class="current-price">36,99 €</span>
The output of the function is this:
product-price pricetext:
stock-price-retry-oos
pricetext:
stock-price-retry
pricetext:
It is not printing class=current-price. Is something wrong with my regex or why does it not find that class??
EDIT:
I found out that the price is not actually inside the HTML of url2. Only the classes that are actually printed out are inside. What's the reason for that and how can I solve that?
The html is not static. It can change over time. If you make a get request to the site's URL you will get the initial value of the html for that site.
But on browsers there is this thing, called javascript, that can make the page's HTML change over time. It's quite common actually:
- The site gets loaded at first with some javascript
- The javascript (developed by the site's creator) than runs and does stuff
- Content dynamically changes by calling some API by that javascript
You can't scrape that content by HTML scraping of the base URL.
If you ask me how I'd do that anyway, is by looking for the site's HTTP requests where it gets the content. Look at that API and use that API myself. Get the data, and store it in some of my servers.
Than on the client I call my server's API to get that data.
Also I'm not really sure that's legal.
But, as far as I understood by your last couple questions, you don't want to do that.
If you really need to do that on the client, you can use WKWebView, load the page, wait for the content to show up, and then get the current HTML of the page by doing something like this:
webView.evaluateJavaScript("document.documentElement.outerHTML.toString()",
completionHandler: { (html: Any?, error: Error?) in
print(html)
})
Look at this answer for more about this.
I hope this solves all of your problem, because I think I don't have much more time to help you :D
I am using AJAX function to update a row. When users click on update button, I call an AJAX function which open a div which contains all the form fields. I have rendered a template and passed that as ajax response. However my form contain a textarea inside which html tags does get escaped e.g link would display like
http://example.com/
I have used JSON.stringify, jQuery.parseJSON before populating div and twig's raw function on form filed but no success.
My form field looks like :
{{ form_widget(form.comments,{ 'value': obj.comments|raw}, {'attr':{'style':'width:'}}) }}
I believe, it has nothing to do with JSON as the template is triggered before encoding the content but I wanted to describe the whole scenario.
Controller
$content = $this->renderView("ex.html.twig", array('form'=>$form->createView()));
return new Response(json_encode($content), 200, array("Content-Type" => "application/json"));
A simple textarea cannot display "HTML" content, so it is always going to show what you call "escaped" content, as it can only display plain text. You need to find an alternative to a textarea, like a wysiwyg editor like TinyMCE or a div with the contentEditable attribute.
use JavaScript escape function after using JSON.stringify
function stringifyP(a) {
return escape(JSON.stringify(a));
}
I'm trying to dynamically load images from a certain post type. I've used the Advanced Custom Fields plugin to attach an image field to my post. I'm currently using this code:
$.getJSON('/?json=get_recent_posts&post_type=slides-verhuur&custom_fields=image', {}, function(data){
console.log(data);
});
However when I run the code, the result JSON I get contains a "custom field" attribute, which have an "image" attribute, but this only contains the value of "80", which is the ID of the image. Is there a way to get the image url instead?
You need to set Return Value as "Image URL" instead of "Image ID" Under Custom Fields group a image field as per image shown below.
$post_response->data['CFS'] = CFS()->get(false, $page->ID, array('format' => 'api'));
use 'api' instead of 'raw'
I'm trying to get JSON from my k2 powered Joomla website. As I understood, I need to add index.php?option=com_k2&id=1&lang=mk&task=category&view=itemlist&format=json to my URL, and then it downloads a joomla.json file. As I know now I need to parse this content but I'm suspicious about the correct format in the joomla.json file.
Here is part of the content of the json file:
{"id":"6","title":"\u0424\u0443\u0441\u0442\u0430\u043d \u041a\u0440\u0438\u0441\u0442\u0438\u043d\u0430","alias":"fustan-kristina","link":"\/katalog\/vencanici\/fustan-kristina.html","catid":"1","introtext":"<ul>\r\n<li>\u041e\u0432\u043e\u0458 \u0444\u0443\u0441\u0442\u0430\u043d \u0435 \u0438\u0437\u0440\u0430\u0431\u043e\u0442\u0435\u043d \u043e\u0434 100% \u0441\u0432\u0438\u043b\u0430.<\/li>\r\n<li>\u041d\u0435\u0436\u0435\u043d \u043c\u0430\u0442\u0435\u0440\u0438\u0458\u0430\u043b<\/li>\r\n<li>\u0414\u043e\u0434\u0430\u0434\u0435\u043d\u0438 \u0446\u0438\u0440\u043a\u043e\u043d\u0438<\/li>\r\n<\/ul>\r\n<p>\u0412\u0438\u0441\u0442\u0438\u043d\u0441\u043a\u0438 \u043d\u0435\u0432\u0435\u0441\u0442\u0438\u043d\u0441\u043a\u0438 \u0444\u0443\u0441\u0442\u0430\u043d<\/p>\r\n<p>\u0411\u0443\u0442\u0438\u043a \u0412\u0438\u043a\u0442\u043e\u0440<\/p>","fulltext":"","extra_fields":[{"id":"1","name":"\u0426\u0435\u043d\u0430","value":"12000 \u0434\u0435\u043d.","type":"textfield","group":"1","published":"1","ordering":"1"},{"id":"6","name":"\u0412\u0435\u0431 \u0421\u0442\u0440\u0430\u043d\u0430","value":"<a href=\"http:\/\/www.domain.mk\" target=\"_blank\">http:\/\/www.domain.mk<\/a>","type":"link","group":"1","published":"1","ordering":"2"}],"created":"2012-07-11 09:42:04","created_by_alias":"","modified":"2012-07-11 11:26:34","featured":"0","image":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_S.jpg","imageWidth":"200","image_caption":"\u0424\u0443\u0441\u0442\u0430\u043d \u041a\u0440\u0438\u0441\u0442\u0438\u043d\u0430","image_credits":"","imageXSmall":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_XS.jpg","imageSmall":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_S.jpg","imageMedium":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_M.jpg","imageLarge":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_L.jpg","imageXLarge":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_XL.jpg","video":null,"video_caption":"","video_credits":"","gallery":null,"hits":"50","category":{"id":"1","name":"\u0412\u0435\u043d\u0447\u0430\u043d\u0438\u0446\u0438","alias":"vencanici","link":"\/katalog\/katalog\/vencanici.html","description":"","image":"","ordering":"1"},"tags":[{"id":"1","name":"kristina","published":"1","link":"\/tag\/kristina.html"},{"id":"2","name":"fustan","published":"1","link":"\/tag\/fustan.html"},{"id":"3","name":"vencanica","published":"1","link":"\/tag\/vencanica.html"}],"attachments":[],"votingPercentage":100,"numOfvotes":"(1 \u0413\u043b\u0430\u0441\u0430\u0458)","author":{"name":"\u0414\u0430\u0440\u043a\u043e \u041f\u0435\u0442\u043a\u043e\u0432\u0441\u043a\u0438","link":"\/blog\/\u0414\u0430\u0440\u043a\u043e-\u041f\u0435\u0442\u043a\u043e\u0432\u0441\u043a\u0438.html","avatar":"http:\/\/www.gravatar.com\/avatar\/bb9f8918a0a63b260d46eb419bf1a894?s=100&default=http%3A%2F%domain.mk%2Fcomponents%2Fcom_k2%2Fimages%2Fplaceholder%2Fuser.png","profile":{"gender":null}},"numOfComments":"0","events":{"BeforeDisplay":"","AfterDisplay":"","AfterDisplayTitle":"","BeforeDisplayContent":"","AfterDisplayContent":"","K2BeforeDisplay":"","K2AfterDisplay":"","K2AfterDisplayTitle":"","K2BeforeDisplayContent":"","K2AfterDisplayContent":"","K2CommentsCounter":""}}]}
Is this json in the correct format,and is this the method that I'm doing correct to get json content from my K2 website?
You are having an extra ] and } at the end of your JSON String.
After removing them, your JSON turns out fine and legit.
{"id":"6","title":"\u0424\u0443\u0441\u0442\u0430\u043d \u041a\u0440\u0438\u0441\u0442\u0438\u043d\u0430","alias":"fustan-kristina","link":"\/katalog\/vencanici\/fustan-kristina.html","catid":"1","introtext":"<ul>\r\n<li>\u041e\u0432\u043e\u0458 \u0444\u0443\u0441\u0442\u0430\u043d \u0435 \u0438\u0437\u0440\u0430\u0431\u043e\u0442\u0435\u043d \u043e\u0434 100% \u0441\u0432\u0438\u043b\u0430.<\/li>\r\n<li>\u041d\u0435\u0436\u0435\u043d \u043c\u0430\u0442\u0435\u0440\u0438\u0458\u0430\u043b<\/li>\r\n<li>\u0414\u043e\u0434\u0430\u0434\u0435\u043d\u0438 \u0446\u0438\u0440\u043a\u043e\u043d\u0438<\/li>\r\n<\/ul>\r\n<p>\u0412\u0438\u0441\u0442\u0438\u043d\u0441\u043a\u0438 \u043d\u0435\u0432\u0435\u0441\u0442\u0438\u043d\u0441\u043a\u0438 \u0444\u0443\u0441\u0442\u0430\u043d<\/p>\r\n<p>\u0411\u0443\u0442\u0438\u043a \u0412\u0438\u043a\u0442\u043e\u0440<\/p>","fulltext":"","extra_fields":[{"id":"1","name":"\u0426\u0435\u043d\u0430","value":"12000 \u0434\u0435\u043d.","type":"textfield","group":"1","published":"1","ordering":"1"},{"id":"6","name":"\u0412\u0435\u0431 \u0421\u0442\u0440\u0430\u043d\u0430","value":"<a href=\"http:\/\/www.domain.mk\" target=\"_blank\">http:\/\/www.domain.mk<\/a>","type":"link","group":"1","published":"1","ordering":"2"}],"created":"2012-07-11 09:42:04","created_by_alias":"","modified":"2012-07-11 11:26:34","featured":"0","image":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_S.jpg","imageWidth":"200","image_caption":"\u0424\u0443\u0441\u0442\u0430\u043d \u041a\u0440\u0438\u0441\u0442\u0438\u043d\u0430","image_credits":"","imageXSmall":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_XS.jpg","imageSmall":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_S.jpg","imageMedium":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_M.jpg","imageLarge":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_L.jpg","imageXLarge":"\/media\/k2\/items\/cache\/ada9a09acea936d776a6f55c82778c43_XL.jpg","video":null,"video_caption":"","video_credits":"","gallery":null,"hits":"50","category":{"id":"1","name":"\u0412\u0435\u043d\u0447\u0430\u043d\u0438\u0446\u0438","alias":"vencanici","link":"\/katalog\/katalog\/vencanici.html","description":"","image":"","ordering":"1"},"tags":[{"id":"1","name":"kristina","published":"1","link":"\/tag\/kristina.html"},{"id":"2","name":"fustan","published":"1","link":"\/tag\/fustan.html"},{"id":"3","name":"vencanica","published":"1","link":"\/tag\/vencanica.html"}],"attachments":[],"votingPercentage":100,"numOfvotes":"(1 \u0413\u043b\u0430\u0441\u0430\u0458)","author":{"name":"\u0414\u0430\u0440\u043a\u043e \u041f\u0435\u0442\u043a\u043e\u0432\u0441\u043a\u0438","link":"\/blog\/\u0414\u0430\u0440\u043a\u043e-\u041f\u0435\u0442\u043a\u043e\u0432\u0441\u043a\u0438.html","avatar":"http:\/\/www.gravatar.com\/avatar\/bb9f8918a0a63b260d46eb419bf1a894?s=100&default=http%3A%2F%domain.mk%2Fcomponents%2Fcom_k2%2Fimages%2Fplaceholder%2Fuser.png","profile":{"gender":null}},"numOfComments":"0","events":{"BeforeDisplay":"","AfterDisplay":"","AfterDisplayTitle":"","BeforeDisplayContent":"","AfterDisplayContent":"","K2BeforeDisplay":"","K2AfterDisplay":"","K2AfterDisplayTitle":"","K2BeforeDisplayContent":"","K2AfterDisplayContent":"","K2CommentsCounter":""}}
Use this site to test your JSON String.
http://jsonlint.com/
If you are having difficulty visualizing your JSON String in 3D, try http://jsonviewer.stack.hu/ You could format the String to visualize the JSON.
I am using &format=json too to bulid my website.
$.get('//url with format=json',function(d){$('.k2ItemTitle').html(d.item.title)})
I don't even have to use $.pareJSON
I want to render some json data using HTML template.
I haven't started implementing anything yet, but I would like to be able to "set" values of data from json to html element which contains template for one record, or to render some collection of items using some argument which is template html for each item, but also to be able to get JSON object back in same format as source JSON which was used to render item (I want my initial JSON to contain some more information about behavior of record row, without the need to make ajax request to check if user can or can't do something with this record, and not all of this info is visible in template).
I know that I could make hidden form with an input element for each property of object to store, and mapper function to/from JSON, but it sounds like overkill to me, and I don't like that, I want some lighter "envelope".
I was wondering is there some JS library that can "serialize" and "deserialize" JSON objects into html so I can store it somewhere in DOM (i.e. in element which contains display for data, but I want to be able to store additional attributes which don't have to be shown as form elements)?
UPDATE As first answer suggested storing JSON in global variable, I also have thought about that, and my "best" mental solution was to make JavaScript module (or jQuery plugin) which would do "mapping" of JSON to html, and if not possible to store values in html then it can store them in internal variable, so when I want to "get" data from html element it can pull it from its local copy. I want to know is there better way for this? If there is some library that stores this info in variable, but does real-time "binding" of that data with html, I would be very happy with that.
UPDATE 2 This is now done using http://knockoutjs.com/, no need to keep json in DOM anymore, knockout does the JSON<=>HTML mapping automatically
Why not store it as nature intended: as a javascript object? The DOM is a horrible place.
That said, jQuery has the data method that allows just this.
So you want to keep a reference to the JSON data that created your DOMFragment from a template?
Let's say you have a template function that takes a template and data and returns a DOM node.
var node = template(tmpl, json);
node.dataset.origJson = json;
node.dataset.templateName = tmpl.name;
You can store the original json on the dataset of a node. You may need a dataset shim though.
There is also no way to "map" JSON to HTML without using a template engine. Even then you would have to store the template name in the json data (as meta data) and that feels ugly to me.
I have done this in the past as well in a couple of different ways.
The $('selector').data idea is probably one of the most useful techniques. I like this way of storing data because I can store the data in a logical, intuitive and orderly fashion.
Let's say you have an ajax call that retrieves 3 articles on page load. The articles may contain data relating to the headline, the date/time, the source etc. Let's further assume you want to show the headlines and when a headline is clicked you want to show the full article and its details.
To illustrate the concept a bit let's say we retrieve json looking something like:
{
articles: [
{
headline: 'headline 1 text',
article: 'article 1 text ...',
source: 'source of the article, where it came from',
date: 'date of the article'
},
{
headline: 'headline 2 text',
article: 'article 2 text ...',
source: 'source of the article, where it came from',
date: 'date of the article'
},
{
headline: 'headline 3 text',
article: 'article 3 text ...',
source: 'source of the article, where it came from',
date: 'date of the article'
}
]
}
From an ajax call like this . . .
$.ajax({
url: "news/getArticles",
data: { count: 3, filter: "popular" },
success: function(data){
// check for successful data call
if(data.success) {
// iterate the retrieved data
for(var i = 0; i < data.articles.length; i++) {
var article = data.articles[i];
// create the headline link with the text on the headline
var $headline = $('<a class="headline">' + article.headline + '</a>');
// assign the data for this article's headline to the `data` property
// of the new headline link
$headline.data.article = article;
// add a click event to the headline link
$headline.click(function() {
var article = $(this).data.article;
// do something with this article data
});
// add the headline to the page
$('#headlines').append($headline);
}
} else {
console.error('getHeadlines failed: ', data);
}
}
});
The idea being we can store associated data to a dom element and access/manipulate/delete that data at a later time when needed. This cuts down on possible additional data calls and effectively caches data to a specific dom element.
anytime after the headline link is added to the document the data can be accessed through a jquery selector. To access the article data for the first headline:
$('#headlines .headline:first()').data.article.headline
$('#headlines .headline:first()').data.article.article
$('#headlines .headline:first()').data.article.source
$('#headlines .headline:first()').data.article.date
Accessing your data through a selector and jquery object is sorta neat.
I don't think there are any libraries that store json in dom.
You could render the html using the data from json and keep a copy of that json variable as a global variable in javascript.