I am trying to extract links from the summary section of a wikipedia page. I tried the below methods :
This url extracts all the links of the Deep learning page:
https://en.wikipedia.org/w/api.php?action=query&prop=links&titles=Deep%20learning
And for extracting links associated to any section I can filter based on the section id - for e.g.,
for the Definition section of same page I can use this url : https://en.wikipedia.org/w/api.php?action=parse&prop=links&page=Deep%20learning§ion=1
for the Overview section of same page I can use this url : https://en.wikipedia.org/w/api.php?action=parse&prop=links&page=Deep%20learning§ion=2
But I am unable to figure out how to extract only the links from summary section
I even tried using pywikibot to extract linkedpages and adjusting plnamespace variable but couldn't get links only for summary section.
You need to use https://en.wikipedia.org/w/api.php?action=parse&prop=links&page=Deep%20learning§ion=0
Note that this also includes links in the
{{machine learning bar}} and {{Artificial intelligence|Approaches}} templates however (to the right of the screen).
You can use Pywikibot with the following commands
>>> import pywikibot
>>> from pwikibot import textlib
>>> site = pywikibot.Site('wikipedia:en') # create a Site object
>>> page = pywikibot.Page(site, 'Deep learning') # create a Page object
>>> sect = textlib.extract_sections(page.text, site) # divide content into sections
>>> links = sorted(link.group('title') for link in pywikibot.link_regex.finditer(sect.head))
Now links is a list containing all link titles in alphabethical order. If you prefer Page objects as result you may create them with
>>> pages = [pywikibot.Page(site, title) for title in links]
It's up to you to create a script with this code snippets.
Related
I would like to get each link that the each boxes contain, the page is https://www.quattroruote.it/listino/audi
In this webpage there are all the model that this brand is producing, and each model is a boxes that links to another page (the one in which I should work with).
My problem is that the initial page do not load all the boxes the first time, you have to scroll down and press the red button "Carica altri modelli" (which mean "Load other models").
Is there a way to automatically store in one variable all the links that i need? For example, the first links of the first box is "/listino/audi/a1"
Thanks in advance to anyone who try to help me!!
Not sure exactly what links you want, but you can make the requests iterating through the itemStart parameter.
import requests
from bs4 import BeautifulSoup
for i in range(1,100):
print('\t\tList start %s' %i)
url = 'https://www.quattroruote.it/listino/ricerca-more-desktop.html'
payload = {
'area': 'NEW',
'itemStart': '%s' %(i*8),
'_': '1634219611449'}
response = requests.get(url, params=payload)
soup = BeautifulSoup(response.text, 'html.parser')
links = soup.find_all('a', href=True)
for link in links:
print(link['href'])
I am trying to create a alert section for a menu (similar to the menu on this page https://metageeky.github.io/mega-menu/responsive-header.html)
*Each alert should have an effective date (date alert is “posted” live) and resolved date (date alert is “removed” live). Each alert will also have a maximum of one to two sentences of text describing situation.
The number of active/current alerts will appear in parenthesis following the icon and ALERT link text.
The icon and text are Dark Orange. When you hover over the icon and text, an underline appears.
When users click on the link, they are taken to a page that lists all active alerts. At bottom of page, message displays “If you are experiencing an issue, please contact us at....”
If there are no Alerts:
The number of alerts in parenthesis following the icon and link text will not appear.
Both the icon and alert text will be Primary Blue.
When Users click on the link, they are taken to a secondary alerts page that displays a message that says “There are currently no active alerts. If you are experiencing an issue, please contact us at...”
How would i achieve this?
Thank you.
There is a lot to unpack in your question but here is a high level approach.
1. Define your model
Read the Django docs on how to create a Model
Read the Django docs on what types of Fields exist
In your models.py, you will need to create a new model that has all the data you need for your requirements.
from django.db import models
class Alert(models.Model):
title = models.CharField()
description = models.TextField()
date_from = models.DateTimeField()
date_to = models.DateTimeField()
2. Ensure you can edit/manage your model data
Now you need to provide a way for your admin users to access the data model, edit & create items.
Wagtail has a great Snippets feature that allows this to work without too many changes, you will need to add #register_snippet on your model and also define some panels.
from wagtail.admin.edit_handlers import FieldPanel
from wagtail.snippets.models import register_snippet
from django.db import models
#register_snippet
class Alert(models.Model):
#... fields (defined above)
panels = [
FieldPanel('title'),
FieldPanel('description'),
FieldPanel('date_from'),
FieldPanel('date_to'),
]
def __str__(self):
return self.title
3. Prepare a template tag to show the queried data
Now you will need to work out how to query the model in a way that it will return the alerts based on your requirements (current date should be within the date range of the data).
Django has docs on writing queries
The simplest way to get the results of this query into the template will be with a custom Template Tag
An inclusion_tag is a way to have a small template fragment that can be used anywhere with custom data (without having to pass it into each View).
In the example below, you will still need to create the template file current_alerts.html which will contain how you want to render the alerts.
In your template tag template you can also use the page_url tag to provide a link to the alerts_page
# template_tags/custom_tags.py
# remember to create a template_tags/__init__.py file also
from django import template
from .models import Alert
register = template.Library()
#register.inclusion_tag('current_alerts.html')
def show_alerts():
# just returns all alerts, but this query can be refined to suit what you need
current_alerts = alerts.Objects.all()
alerts_page = AlertPage.Objects.all().first() # this assumes there will only ever be one
return {'alerts_page',alerts_page,'current_alerts': current_alerts}
4. Use your template tag & add styling
Now you need to include the tag at the top of the page inside your root/shared template.
{% extends "base.html" %}
{% load custom_tags %}
{% block body_class %}template-blogpage{% endblock %}
{% show_alerts %}
{% block content %}...{% endblock %}
5. Create a AlertsPage
You will need to create a new Page type to redirect users to within your alerts link.
https://docs.wagtail.io/en/stable/topics/pages.html
This Page can be anywhere in your tree and the Page's view template can also use the same shared template or you can pass the alerts to the view via the template context
How do I get a clickable "permalink" to show on hover (for e.g. h2 headers) similar to this.
I'm using the pelican framework and am writing in restructuredText. I can't figure out where in the stack to tweak to enable this (pelican? rst? jinja2?)
It's called a label. Usage is documented under the ref role.
To support cross-referencing to arbitrary locations in any document, the standard reST labels are used. For this to work label names must be unique throughout the entire documentation. There are two ways in which you can refer to labels:
If you place a label directly before a section title, you can reference to it with :ref:label-name. For example:
.. _my-reference-label:
Section to cross-reference
--------------------------
This is the text of the section.
It refers to the section itself, see :ref:`my-reference-label`.
Edit
For Pelican, there is a plugin called headerid that will render permalinks.
You can get that through the options of the Markdown TOC extension.
Just set the permalink option to True in your pelicanconf.py:
MARKDOWN = {
'extension_configs': {
'markdown.extensions.toc': {'permalink': 'True'}
}
}
I have seen great examples of how bokeh allows you to hover over a data point and display pop up details for it. There are cases the details is so overwhelming voluminous, it really requires a side panel to display it all. Is bokeh a complete enough widget toolkit where I can create a side panel to the main display and show details of a data point following the cursor?
Can someone point out some sample code, or at least the relevant api's.
If you prefer a higher-level API for building and linking Bokeh-based plots, you can use HoloViews; see linking examples at http://holoviews.org/reference/index.html#streams and instructions at http://holoviews.org/user_guide/Custom_Interactivity.html . For example:
import param, numpy as np, holoviews as hv
from holoviews import opts, streams
hv.extension('bokeh')
xvals = np.linspace(0,4,202)
ys,xs = np.meshgrid(xvals, -xvals[::-1])
img = hv.Image(np.sin(((ys)**3)*xs))
pointer = streams.PointerXY(x=0,y=0, source=img)
dmap = hv.DynamicMap(lambda x, y: hv.Points([(x, y)]), streams=[pointer])
dmap = dmap.redim.range(x=(-0.5,0.5), y=(-0.5,0.5))
img + dmap.opts(size=10)
You can find many examples on https://docs.bokeh.org . What you want is possible by adding a callback and updating the relevant part. In this example the div is what you name a side panel in your question.
#for bokeh 1.0.4
from bokeh.plotting import figure
from bokeh.models import ColumnDataSource,Div,Row
from bokeh.io import curdoc
from bokeh.events import Tap
#the data
d={'x':[1,2],'y':[3,4],'info':['some information on a first datapoint','some information on a second datapoint']}
source=ColumnDataSource(d)
tooltips = [("x", "$x"),("y", "$y"),("info","#info")]
fig=figure(tools="tap,reset",tooltips=tooltips)
c=fig.circle('x','y',source=source,size=15)
def callback(event):
indexActive=source.selected.indices[0]
layout.children[1]=Div(text=d['info'][indexActive])#adjust the info on the right
fig.on_event(Tap, callback)
div=Div(text=d['info'][0])
layout=Row(fig,div)
curdoc().add_root(layout)
To run this code, save it as code.py, open a cmd and type "bokeh serve code.py --show".
In pelican, by default, blog articles are listed on the index.html file.
What I want instead is that I use a static page as my home page and put all the blog articles on a dedicated "Blog" page.
How can I get this done?
While there are several possible methods for achieving your desired goals, I would start with the following changes to your settings file:
SITEURL = '/blog'
OUTPUT_PATH = 'output/blog'
PAGE_URL = '../{slug}.html'
PAGE_SAVE_AS = '../{slug}.html'
DISPLAY_PAGES_ON_MENU = False
DISPLAY_CATEGORIES_ON_MENU = False
MENUITEMS = [('Home', '/'), ('Blog', '/blog/')]
Put your blog posts in content/ as usual, and then create your home page with the following headers and save as content/pages/home.md:
Title: Home
URL: ../
Save_as: ../index.html
This is the home page.
Caveats:
Dynamic navigation menu generation has been effectively turned off since it doesn't work well with this configuration. Highlighting for the currently-active menu item — a feature you normally get out-of-the-box — will not be present in this configuration and, if desired, must be implemented separately in your theme.
If your theme's base.html template has a link to your site home that depends on SITEURL (e.g., as the notmyidea theme does), you will need to change the link to point to <a href="/"> instead.
Set the following in the pelicanconf
DIRECT_TEMPLATES = ['blog']
PAGINATED_DIRECT_TEMPLATES = ['blog']
1st line will set blog.html for the articles
2nd line will allow pagination of blog.html file
For the index page, create a pages folder in the content directory and create the .md file there and set save_as:index.html this will save the md file as index.html
This is covered in the Pelican FAQ
- "How can I override the generated URL of a specific page or article?"
Basically, in your contents folder, create two subfolders:
/contents/blogs, which will store all your blog entries
/content/pages, which will store your other static pages (including your home page)
In the pages subfolder, create a file (e.g. home.rst) with the option :save_as: index.html, which will make this file your home page. E.g.:
Home
####
:date: 2015-05-22 12:30
:url:
:save_as: index.html
This is my home page
In your pelicanconf.py file, specify the following options:
DISPLAY_PAGES_ON_MENU = False
DISPLAY_CATEGORIES_ON_MENU = True
USE_FOLDER_AS_CATEGORY = True
PATH = 'content'
ARTICLE_PATHS = ['articles',]
PAGE_PATHS = ['pages',]
MENUITEMS = ()
You should now have a home page and a contents bar with a Blogs menu.
If you want to add more menus to the contents bar (for example an About or CV menu), create the corresponding files in your pages folder, and add them to MENUITEMS:
MENUITEMS = (
('About', '/pages/about.html'),
('CV', '/pages/cv.html'),
)
I have an answer similar to the one Justin Mayer gave, except in mine I change blog article urls instead of page urls.
I've been getting the following error when trying to use the answer above, so it might be useful to other people having the same issue
ERROR: Skipping volunteering.rst: file '../volunteering.html' would be written outside output path
ERROR: Skipping presentations.rst: file '../presentations.html' would be written outside output path
Make all article urls to be under 'blog/' url
ARTICLE_URL = "blog/{date:%Y}-{date:%m}-{date:%d}-{slug}.html"
ARTICLE_SAVE_AS = "blog/{date:%Y}-{date:%m}-{date:%d}-{slug}.html"
Put blog index under 'blog/' url
INDEX_SAVE_AS = "blog/index.html"
Add a explicit menu item for blog index
MENUITEMS = [
('home', '/'),
('blog', '/blog'),
]
As your page is now an index page, automatically displaying link to that page in the menu will lead to a broken link, so you will have to set the following option and specify the following flag
DISPLAY_PAGES_ON_MENU = False
For the new index page, add a directive save_as, like Justin Mayer pointed it out. Here how it looks in rst
About
=====
:slug: about
:category: About
:save_as: index.html
This should give you a home page and an index page for articles.
When you want to add more static pages, you will also need to add them in menu items that still contains '/pages' prefix in the url if you want links to the pages appear in a menu. i.e for the volunteering.rst with the following content,
Volunteering
============
:slug: about
:category: About
Your MENUITEMS variable will look like the following
MENUITEMS = [
('home', '/'),
('blog', '/blog'),
('volunteering', '/pages/volunteering'),
]
I tested this answer on pelican 4.2.0.
You can use the following settings to put the index file for example at /blog/index.html.
INDEX_SAVE_AS = 'blog/index.html'
INDEX_URL = 'blog/'
Then you created a home.md page and use "save_as: index.html" directive for the actual home page.