Converting the output of MediaWiki to plain text - json

Question

Using the MediaWiki API, this gives me an output like so, for search term Tiger
https://simple.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Tiger&format=json&exintro=1
Response:
{"batchcomplete":"","query":{"pages":{"9796":{"pageid":9796,"ns":0,"title":"Tiger","extract":"<p>The <b>tiger</b> (<i>Panthera tigris</i>) is a carnivorous mammal. It is the largest living member of the cat family, the Felidae. It lives in Asia, mainly India, Bhutan, China and Siberia.</p>\n<p></p>"}}}}
How do I get an output as
The tiger (Panthera tigris) is a carnivorous mammal. It is the largest living member of the cat family, the Felidae. It lives in Asia, mainly India, Bhutan, China and Siberia.
Please can someone also tell me how to store everything in a text file? I'm a beginner here so please be nice. I need this for a project I'm doing in Bash, on a Raspberry Pi 2, with Raspbian

Answer 1

It's usually recommended to use JSON parser for handling JSON, one that I like is jq
% jq -r '.query.pages[].extract' file
<p>The <b>tiger</b> (<i>Panthera tigris</i>) is a carnivorous mammal. It is the largest living member of the cat family, the Felidae. It lives in Asia, mainly India, Bhutan, China and Siberia.</p>
<p></p>
To remove the HTML tags you can do something like:
... | sed 's/<[^>]*>//g'
Which will remove HTML tags that are not on continues lines:
% jq -r '.query.pages[].extract' file | sed 's/<[^>]*>//g'
The tiger (Panthera tigris) is a carnivorous mammal. It is the largest living member of the cat family, the Felidae. It lives in Asia, mainly India, Bhutan, China and Siberia.
file is the file the JSON is stored in, eg:
curl -so - 'https://simple.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Tiger&format=json&exintro=1' > file
jq '...' file
or
jq '...' <(curl -so - 'https://simple.wikipedia.org/w/api.php?action=query&prop=extracts&titles=Tiger&format=json&exintro=1')
You can install jq with:
sudo apt-get install jq
For your example input you can also use grep with -P (PCRE). But using a proper JSON parser as above is recommended
grep -oP '(?<=extract":").*?(?=(?<!\\)")' file
<p>The <b>tiger</b> (<i>Panthera tigris</i>) is a carnivorous mammal. It is the largest living member of the cat family, the Felidae. It lives in Asia, mainly India, Bhutan, China and Siberia.</p>\n<p></p>

Answer 2

If you're using PHP, you can do it fairly easily such as below.
Accessing the text
We know that the text is stored inside the extract property, so we need to access that.
The easiest way to do this would be to parse the string from the API into an object format, which is done with the json_decode method in PHP. You can then access the extract property from that object, and this will give you your string. The code would be something like this:
//Get the string from the API, however you've already done it
$JSONString = getFromAPI();
//Use the inbuilt method to create a JSON object
$JSONObject = json_decode($JSONString);
//Follow the structure to get the pages property
$Pages = JSONObject->query->pages;
//Here, we don't know what the Page ID is (because the MediaWiki API returns a different number, depending on the page)
//Therefore we need to simply get the first key, and within it should be our desired 'extract' key
$Extract = "";
foreach($Pages as $value) {
$Extract = $value->extract;
break;
}
//$Extract now contains our desired text
Writing it to a file
Now we need to write the contents of $Extract to a file, as you mentioned. This can be done as follows, by utilizing the file_put_contents method.
//Can be anything you want
$file = 'APIResult.txt';
// Write the contents to the file,
// using the LOCK_EX flag to prevent anyone else writing to the file at the same time
file_put_contents($file, $Extract, LOCK_EX);
Aaand we're done!
Documentation
The documentation for these functions (json_decode and file_put_contents) can be found at:
http://php.net/manual/en/function.json-decode.php
http://php.net/manual/en/function.file-put-contents.php

Answer 3

You may find pandoc helpful, from http://pandoc.org/ - it understands a number of file formats on input including Mediawiki, and also has a bunch of file formats on output including plain text. It's more the "Swiss army knife" approach, and since Mediawiki is arbitrarily complicated to parse, you'll want to use something like this that's been through a big test suite.