Web scraping giving different responses - html

I have this dart code, but it gives different outputs everytime i run it
Steps to reproduce
Add both files to same directory
Run
dart pub get
dart code.dart
It will create ten files having different contents
Create a file named code.dart
import "dart:io";
import 'dart:convert';
import 'package:http/http.dart' as http;
import 'package:html/parser.dart';
Future<String> scrapLyrics(String lyricsUrl) async {
final response = await http.get(Uri.parse(lyricsUrl));
var document = parse(response.body);
return document.getElementsByClassName('lyrics')[0].text;
}
main() async {
String ly;
for (int i = 1; i <=10; i++) {
print("Request number $i");
try{
ly= await scrapLyrics("https://www.genius.com/The-chainsmokers-closer-lyrics");
}
catch(e){
ly=e.toString();
}
File file = File("file$i.txt");
await file.writeAsString("$ly");
ly="";
}
}
Create a file pubspec.yaml
name: my_app
environment:
sdk: '>=2.10.0 <3.0.0'
dependencies:
http: ^0.13.3
html: ^0.15.0
Obtained Output(in some cases):
RangeError (index): Invalid value: Valid value range is empty: 0

The class name you're looking for is "Lyrics__Container-sc-1ynbvzw-8" not "lyrics"

Related

Not able to download json file through url (Flutter/dart)

I'd like to download the JSON file which can be accessed by this url https://firebasestorage.googleapis.com/v0/b/prufungssimulation-caff9.appspot.com/o/5qPfjoZjdNXyN5Ob2q3pFbj5MKy1.json?alt=media. Unfortunately I was not able to download this file with the http get-function. I assume that I have some problem with the parsing since if I remove "?alt=media" it works perfectly, but unfortunately I only get the meta-json file. This is the code which I use:
import 'package:http/http.dart' as http;
[...]
[...]
void testFunction(){
var url_string = "https://firebasestorage.googleapis.com/v0/b/prufungssimulation-caff9.appspot.com/o/5qPfjoZjdNXyN5Ob2q3pFbj5MKy1.json?alt=media";
var result = await getJsonFromFirebaseRestAPI(url_string);
print(result);
}
Future<String> getJsonFromFirebaseRestAPI(String url) async {
http.Response response = await http.get(Uri.parse(url));
return response.body;
}
The JSON file result should look like this:
routes: []
It's basically empty I'm just trying to implement the function.
I use http: ^0.13.5
I get the following error:
Error: XMLHttpRequest error.
dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 299:10
createErrorWithStack
dart-sdk/lib/_internal/js_dev_runtime/patch/core_patch.dart 341:28
_throw
dart-sdk/lib/core/errors.dart 116:5
throwWithStackTrace
dart-sdk/lib/async/zone.dart 1378:11
callback
dart-sdk/lib/async/schedule_microtask.dart 40:11
_microtaskLoop
dart-sdk/lib/async/schedule_microtask.dart 49:5
_startMicrotaskLoop
dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 166:15
<fn>
Can anybody solve my issue?
Appreciate your time :)
I did few changes on your code and it is working. There was no issue with the get function. here is the updated piece
import 'package:http/http.dart' as http;
// just a normal main to test the function in terminal
void main(List<String> arguments) async {
await testFunction();
}
// changed this to future and made it async
Future<void> testFunction() async {
var url_string = "https://firebasestorage.googleapis.com/v0/b/prufungssimulation-caff9.appspot.com/o/5qPfjoZjdNXyN5Ob2q3pFbj5MKy1.json?alt=media";
var result = await getJsonFromFirebaseRestAPI(url_string);
print(result);
}
// no changes applied here.
Future<String> getJsonFromFirebaseRestAPI(String url) async {
http.Response response = await http.get(Uri.parse(url));
return response.body;
}

Wp api for flutter

I am trying to use the wordpress api, but it throws me an error when entering the url.
Error: The argument type "String" can't be assigned to the parameter type Uri
Could someone explain the error to me and tell me how the code should look? Thanks
import 'package:http/http.dart' as http;
import 'dart:convert';
Future<List> blog() async {
final response = await http.get(('https://bauk.blog/wp-json/wp/v2/posts?_embed': {"Accept": "application/json"}));
var convertirajson = jsonDecode(response.body);
return convertirajson;
}
You need to parse the url before passing it to http.get()
For that declare your url variable like this:
var url = Uri.parse('https://bauk.blog/wp-json/wp/v2/posts?_embed');
And then pass it to http.get() like this
http.get((url: {...} ));

Flutter local json

I copy this file in my local and try to parse it.
The following are my questions in mind:
1) how to parse the categories [sport, maths] for listview purpose/
2) how to parse item inside the category?
3) does it need to change the format of the json to have simpler codes?
Currently, this is the code
Future<dynamic> _future;
Future<String> _getJson() async {
var response = await rootBundle.loadString('assets/example_2.json');
var decodedJason = json.decode(response);
return (decodedJason); }
void initState() {
_future = _getJson();
super.initState(); }
Thanks in advance
Add Your JSON File To The pubspec.yaml
assets:
- assets/example_2.json
And Then You Can Use rootBundle To Load & Display It
import 'dart:async' show Future;
import 'package:flutter/services.dart' show rootBundle;
Future<String> loadAsset() async {
return await rootBundle.loadString('assets/config.json');
}
Make A Sure Json File Is Not Empty?

fetch json array from webservice

I am trying to fetch a json array from webservice.
The docs say:
_getIPAddress() {
final url = 'https://httpbin.org/ip';
HttpRequest.request(url).then((value) {
print(json.decode(value.responseText)['origin']);
}).catchError((error) => print(error));
}
If I use this code I get the error:
The method request is not defined for the class 'HttpRequest'
While If I try to import:
import 'dart:html';
I get this error:
Target of URI doesn't exist 'dart:html'
For http requests i recommend the http package.
Then after importing the http package you can use it like that for example:
import 'package:http/http.dart' as http;
_getIPAddress() async {
final url = 'https://httpbin.org/ip';
try {
http.Response res = await http.get(url);
print(json.decode(res.body));
} catch(e) {
print(e);
}
}

Ionic2 and get Json

I am trying to use Ionic2 and I made a service to fetch a local stored Json.
import {Injectable} from 'angular2/core';
import {Http, Response} from 'angular2/http';
import {Observable} from 'rxjs/Rx';
import 'rxjs/add/operator/map';
#Injectable()
export class Page1Service {
public constructor(private _http: Http) {}
public GetItems() {
return this._http.get('/app/Ressources/Items.json').map((response: Response) => response.json().data);
}
public PrintJson():boolean {
var myresult;
this.GetItems().subscribe((result) => {
myresult = result;
console.log(result);
});
}
I also a made PrintJson() method that just print the json for test purpose.I got the error:
GET http://localhost:8100/app/Ressources/slides.json 404 (Not Found)
I don't get why. And I can't find an easy and uptodate tutorial. Or should I use fetch()?
First copy your json to the following dir(you can create the folder "data"):
[appname]/www/data/data.json
Type in the following command in your console:
ionic g provider JsonData
It should create a provider for you.Go to that page and enter the following in load() function:
load() {
if (this.data) {
// already loaded data
return Promise.resolve(this.data);
}
// don't have the data yet
return new Promise(resolve => {
// We're using Angular Http provider to request the data,
// then on the response it'll map the JSON data to a parsed JS object.
// Next we process the data and resolve the promise with the new data.
this.http.get('data/data.json').subscribe(res => {
// we've got back the raw data, now generate the core schedule data
// and save the data for later reference
this.data = res.json();
resolve(this.data);
console.log(this.data);
});
});
}
I usually create an Observable wrapped around the api-call like this:
public GetItems() {
return Observable.create(observer => {
this._http.get('/app/Ressources/Items.json').map(res =>res.json()).subscribe(data=>{
observer.next(data)
observer.complete();
});
});
}
Then I have to subscribe on that method in order to get the results and do something with it. (You could be to delegate the result to a list in the GUI)
GetItems().subscribe(data=>{
myResult = data;
});
EDIT: It might help to put this in the class as well
export class MyClass{
static get parameters(){
return [[Http]];
}
}
Just try to get the response.json() rather than response.json().data in GetItems() method
The issue is because of different paths of json files in local browser(computer) and device (android). Create data folder inside the src\assets folder. Move your json file into that.
When we run ionic serve, it will move that folder (with file) into www\assets folder. Then do following things:
Import Platform service of ionic2
import { Platform } from 'ionic-angular';
Inject Platform Service.
constructor(private http: Http, private platform: Platform ) { }
Use Platform Service.
public getItems() {
var url = 'assets/data/Items.json';
if (this.platform.is('cordova') && this.platform.is('android')) {
url = "/android_asset/www/" + url;
}
return this.http.get(url)
.map((res) => {
return res.json()
});
}