I have a website that shows the user the population for the country in an attractive way.
My problem is that I will need to get the data to be updated automatically.
My question is how do I get the source code of the population from this website.
https://www.worldometers.info/world-population/afghanistan-population/
You can use the following free API to get the current population of the country. Then you can use JavaScript to fetch the value and output it on your website. To always display the latest value, you could wrap the fetch method in an interval, which repeats after a given time.
The .CSV file supplied has 225146 records and has the following fields:
FLOW_ID – The unique identifier of the traffic flow measured;
LINK_ID – The unique identifier of the road link being measured;
LINK_DESC – A description of the road link being measured;
FLOW_DIR – The direction of the traffic flow;
PERIOD_TYPE – The type of period when traffic was measured;
F00 to F33 – The traffic flow measured at each hour, i.e. 0:00, 1:00….23:00
this is the main table and split it to make it simple to use
I Split into three tables
Flow_Dim (flow_id(pk), Link_Desc, Flow_Dir) [with grouping]
Period_Dim (p_id(pk), period_type) [just got three values with group by (grouping)]
Traffic_Flow_Fact (flow_id(fk), p_id(fk), F00 to F23 then all the 24 columns of hours) [insert the following query to make it simple]
INSERT INTO `Traffic_Flow_Fact`
(`flow_id`, `period_id`, `H0`, `H1`, `H2`, `H3`)
SELECT junk.`FLOW_ID`, p.`p_id`, SUM(junk.`F00`),
SUM(junk.`F01`), SUM(junk.`F02`),
SUM(junk.`F03`)
FROM TempJunk junk, Period_Dim p
WHERE p.`PERIOD_TYPE` = junk.`PERIOD_TYPE`
GROUP BY p.`PERIOD_TYPE`, junk.`FLOW_ID`, p.`p_id`
Following is the table Traffic_Flow_Fact
I just want to know is there any other way to make hours and there traffic flow value in separate table so that I can find out following solutions -
a) What are the top 5 busiest links during hours public holidays, school holidays and normal time (school term)? What is the average traffic on those links during those periods?
b) What are the 3 least busy links during hours public holidays, school holidays and normal time (school term)?
c) What are the peak time hours for Monash Freeway links? What about Princes Way links?
I am not expecting any answers of these questions these just for reference so that you guys will what needs to be done
I'm trying to scrape the price from the first ticket here page using this xpath:
'.//*[#class="price"]/text()'
This works in the developer's console, but not when I run it in the scrapy shell using response.xpath. I have also tried to following in the shell:
'.//*[#class="initial"]/div[#class="price"]/text()'
and
'//*[#id="tVB901769989"]/div[1]/div[4]' (although I don't think that the id property can be used in the shell like this).
Is there something wrong with the xpaths that I've used, or is there some thing different with the way the page works? Any help would be appreciated. Thanks!
this happens because you are checking at different requests, the page you see doesn't have the information you need inside that file, but it gets it dynamically, in this case from: www.vividseats.com/javascript/tickets.shtml?productionId=1771684
There you can check the prices on json format, I think this is for one item:
{
"s":"Section 111",
"r":"8-22",
"q":"4",
"p":"692.00",
"i":"VB782041491",
"d":"111",
"n":"Zone Seating. The seller is committing to procure these tickets for you upon receipt of your order. After you place your order and your order is confirmed, we guarantee that your tickets will be within the listed zone
or section listed or one comparable and that you will receive these tickets in time for the event or
your money back. Orders exceeding four tickets may be split up into different rows within the requested
zone or section.",
"f":"0",
"l":"Section 111",
"g":"0",
"e":"0",
"h":"07/21/15",
"t":"0",
"v":"",
"c":"84352",
"z":"1",
"rhdn":"0",
"ind":"0",
"sd":"0"
}
where p contains the price.
I am looking for a way to get the Prices and Titles of products using the Amazon's API Operation -> ItemSearch.
So far, I can get the Price using the ResponseGroup -> Offers.
1) Is there any way to get a product title without the API, using the ASIN? For example some kind of url amazon.com/titles/id
2) If not, how can I get the Title without increesing the filesize of the xml returned too much?
1) According to the ASIN wikipedia page, the shortest format URL you can create would be http://amzn.com/ASIN with 'ASIN' being the actual number. If you wanted to parse the HTML returned from a link to find the title, that is another possible option.
2) Add the ItemAttributes response group; you can comma-delimit response groups too so ResponseGroup -> Offers,ItemAttributes is what I'd suggest.
And if ItemAttributes increases filesize too dramatically, you can find a list of valid response groups on the ItemSearch operation page in the docs.
Wow, makes your head spin!
I am about to start a project, and although my mySql is OK, I can't get my head around what required for this:
I have a table of web addresses.
id,url
1,http://www.url1.com
2,http://www.url2.com
3,http://www.url3.com
4,http://www.url4.com
I have a table of users.
id,name
1,fred bloggs
2,john bloggs
3,amy bloggs
I have a table of categories.
id,name
1,science
2,tech
3,adult
4,stackoverflow
I have a table of categories the user likes as numerical ref relating to the category unique ref. For example:
user,category
1,4
1,6
1,7
1,10
2,3
2,4
3,5
.
.
.
I have a table of scores relating to each website address. When a user visits one of these sites and says they like it, it's stored like so:
url_ref,category
4,2
4,3
4,6
4,2
4,3
5,2
5,3
.
.
.
So based on the above data, URL 4 would score (in it's own right) as follows: 2=2 3=2 6=1
What I was hoping to do was pick out a random URL from over 2,000,000 records based on the current users interests.
So if the logged in user likes categories 1,2,3 then I would like to ORDER BY a score generated based on their interest.
If the logged in user likes categories 2 3 and 6 then the total score would be 5. However, if the current logged in user only like categories 2 and 6, the URL score would be 3. So the order by would be in context of the logged in users interests.
Think of stumbleupon.
I was thinking of using a set of VIEWS to help with sub queries.
I'm guessing that all 2,000,000 records will need to be looked at and based on the id of the url it will look to see what scores it has based on each selected category of the current user.
So we need to know the user ID and this gets passed into the query as a constant from the start.
Ain't got a clue!
Chris Denman
What I was hoping to do was pick out a random URL from over 2,000,000 records based on the current users interests.
This screams for predictive modeling, something you probably wouldn't be able to pull off in the database. Basically, you'd want to precalculate your score for a given interest (or more likely set of interests) / URL combination, and then query based on the precalculated values. You'd most likely be best off doing this in application code somewhere.
Since you're trying to guess whether a user will like or dislike a link based on what you know about them, Bayes seems like a good starting point (sorry for the wikipedia link, but without knowing your programming language this is probably the best place to start): Naive Bayes Classifier
edit
The basic idea here is that you continually run your precalculation process, and once you have enough data you can try to distill it to a simple formula that you can use in your query. As you collect more data, you continue to run the precalculation process and use the expanded results to refine your formula. This gets really interesting if you have the means to suggest a link, then find out whether the user liked it or not, as you can use this feedback loop really improve the prediction algorithm (have a read on machine learning, particularly genetic algorithms, for more on this)
I did this in the end:
$dbh = new NewSys::mySqlAccess("xxxxxxxxxx","xxxxxxxxxx","xxxxxxxxx","localhost");
$icat{1}='animals pets';
$icat{2}='gadget addict';
$icat{3}='games online play';
$icat{4}='painting art';
$icat{5}='graphic designer design';
$icat{6}='philosophy';
$icat{7}='strange unusual bizarre';
$icat{8}='health fitness';
$icat{9}='photography photographer';
$icat{10}='reading books';
$icat{11}='humour humor comedy comedian funny';
$icat{12}='psychology psychologist';
$icat{13}='cartoons cartoonist';
$icat{14}='internet technology';
$icat{15}='science scientist';
$icat{16}='clothing fashion';
$icat{17}='movies movie latest';
$icat{18}="\"self improvement\"";
$icat{19}='drawing art';
$icat{20}='latest band member';
$icat{21}='shop prices';
$icat{22}='recipe recipes food';
$icat{23}='mythology';
$icat{24}='holiday resorts destinations';
$icat{25}="(rude words)";
$icat{26}="www website";
$dbh->Sql("DELETE FROM precalc WHERE member = '$fdat{cred_id}'");
$dbh->Sql("SELECT * FROM prefs WHERE member = '$fdat{cred_id}'");
#chos=();
while($dbh->FetchRow()){
$cat=$dbh->Data('category');
$cats{$cat}='#';
}
foreach $cat (keys %cats){
push #chos,"\'$cat\'";
push #strings,$icat{$cat};
}
$sqll=join("\,",#chos);
$words=join(" ",#strings);
$dbh->Sql("select users.id,users.url,IFNULL((select sum(scoretot.scr) from scoretot where scoretot.id = users.id and scoretot.category IN \($sqll\)),0) as score from users WHERE MATCH (description,lasttweet) AGAINST ('$words' IN BOOLEAN MODE) AND IFNULL((SELECT ref FROM visited WHERE member = '$fdat{cred_id}' AND user = users.id LIMIT 1),0) = 0 ORDER BY score DESC limit 30");
$cnt=0;
while($dbh->FetchRow()){
$id=$dbh->Data('id');
$url=$dbh->Data('url');
$score=$dbh->Data('score');
$dbh2->Sql("INSERT INTO precalc (member,user,url,score) VALUES ('$fdat{cred_id}','$id','$url','$score')");
$cnt++;
}
I came up with this answer about three months ago, and just cannot read it. So sorry, I can't explain how it finally worked, but it managed to query 2 million websites and choose one based on the history of a users past votes on other sites.
Once I got it working, I moved on to another problem!
http://www.staggerupon.com is where it all happens!
Chris