I am trying to download a .csv file from a website using a .bat file in the current folder (or specified).
I think there is some issue with how the download works for this .csv file; I think it downloads only on button/tab click.
If you scroll down there is the option - Download this time series and green color tab .csv with the following uri:
https://www.ons.gov.uk/generator?format=csv&uri=/economy/inflationandpriceindices/timeseries/chaw/mm23
This is my .bat file content:
#echo off
SET "FILENAME=%~dp0\series.csv"
bitsadmin.exe /transfer "JobName" https://www.ons.gov.uk/generator?format=csv&uri=/economy/inflationandpriceindices/timeseries/chaw/mm23 "%FILENAME%"
I have even tried to do this using XSLT 2.0, but I was unable to automate this download.
I'm not so familiar with batch and I think powershell is much better tool for this job in terms of long time support and easiness. Here is very sample script to download file by given link
$url = "https://www.ons.gov.uk/generator?format=csv&uri=/economy/inflationandpriceindices/timeseries/chaw/mm23"
$output = "$PSScriptRoot\series.csv"
$start_time = Get-Date
Invoke-WebRequest -Uri $url -OutFile $output
Write-Output "Time taken: $((Get-Date).Subtract($start_time).Seconds) second(s)"
Create sample file with script named like download.ps1.
You'll need to cd to file directory and run it like .\download.ps1
Related
SQL Server 2019. In the SSRS Report Server web interface I can upload any file, Pdf, Excel, Word, etc. I've got a lot of files I want to upload and the web interface only allows me to do one at a time. Can I upload all files in a folder to the SSRS server using Powershell? So far what I've found only seems to work for SSRS files - rdl, rsd, etc. Is there some other way to upload multiple non-SSRS files? Thanks!
You can use the below PowerShell script, you will need to change the folder location and the report server URL as well as the -RSFolder reference. The script will upload all files within the folder. Please be aware that SSRS does restrict some file types, these can be found using the following SQL code :
SELECT ConfigInfoID
,Name
,Value
FROM ReportServer.dbo.ConfigurationInfo
WHERE Name = 'AllowedResourceExtensionsForUpload'
-- PowerShell Script --
$FileLocation = "C:\Files\"
$Files = Get-ChildItem $FileLocation
foreach ($File in $Files)
{
Write-RSRestCatalogItem -Overwrite -ReportPortalUri http://ReportServer/Reports/ -Path $Files.FullName -RsFolder "Files" -RestApiVersion V1.0
}
Also should have mentioned that you need to install a PowerShell module:
Install-Module -Name ReportingServicesTools
I have a html file. So now I wanted to open the html report and read particular line and also the result and save that value to a json file through a jenkins job. have shell script to read whole file line by line. But how can i read a particular line which has a word APPLE in it and save it to a json file.
#!/bin/bash
filename="data.html"
while read -r line
do
name="$line"
echo "Name read from file - $name"
done < "$filename"
This will echo whole page. But i need to search for a particular line having word APPLE.
Please help
I need to format a large JSON file for readability, but every resource I've found (mostly online) doesn't deal with data say, above 1-2 MB. I need to format about 30 MB. Is there any way to do this, or any way to code something to do this?
With python >= 2.6 you can do the following:
For Mac/Linux users:
cat ugly.json | python -mjson.tool > pretty.json
For Windows users (thanks to the comment from dnk.nitro):
type ugly.json | python -mjson.tool > pretty.json
jq can format or beautify a ~100MB JSON file in a few seconds:
jq '.' myLargeUnformattedFile.json > myLargeBeautifiedFile.json
The command above will beautify a single-line ~120MB file in ~10 seconds, and jq gives you a lot of json manipulation capabilities beyond simple formatting, see their tutorials.
jsonpps is the only one worked for me (https://github.com/bazaarvoice/jsonpps).
It doesn't load everything to RAM unlike jq, jsonpp and others that I tried.
Some useful tips regarding installation and usage:
Download url: https://repo1.maven.org/maven2/com/bazaarvoice/jsonpps/jsonpps/1.1/jsonpps-1.1.jar
Shortcut (for Windows):
Create file jsonpps.cmd in the same directory with the following content:
#echo off
java -Xms64m -Xmx64m -jar %~dp0\jsonpps-1.1.jar %*
Shortcut usage examples:
Format stdin to stdout:
echo { "x": 1 } | jsonpps
Format stdin to file
echo { "x": 1 } | jsonpps -o output.json
Format file to file:
jsonpps input.json -o output.json
Background-- I was trying to format a huge json file ~89mb on VS Code using the command (Alt+Shift+F) but the usuals, it crashed. I used jq to format my file and store it in another file.
A windows 11 use case is shown below.
step 1- download jq from the official site for your respective OS - https://stedolan.github.io/jq/
step 2- create a folder in the C drive named jq and paste the executable file that you downloaded into the folder. Rename the file as jq (Error1: beware the file is by default an exe file so do not save it as 'jq.exe' save it only as 'jq')
step 3- set your path variable to the URL of the executable file.
step 4- open your directory on cmd where the json file is stored and type the following command - jq . currentfilename.json > targetfilename.json
replace currentfilename with the file name that you want to format
replace targetfilename with the final file name that you want your data formatted in
within seconds you should see your target file in the same directory in a formatted version which can now be opened on VS Code or any editor for that matter. Any error related to the recognizability of jq as a command can be traced back with high probability to Error 1.
jq jquery json data-preprocessing data-cleaning
You can use Notepad++ (https://notepad-plus-plus.org/downloads/) for formatting large JSON files (tested in Windows).
Install Notepad++
Go to Plugins -> Plugins Admin -> Install the 'Json Viewer' plugin. The plugin source code is present in https://github.com/kapilratnani/JSON-Viewer
After plugin installation, go to Plugins -> JSON Viewer -> Format JSON.
This will format your JSON file
I want to record the voice using html5 and I have tried jRecorder-jQuery too. From the document, it mentioned that the binary file is saved in browser cache. My question is where it? I have checked Chrome's cache but cannot see the temp file.
host (Mandatory): The PHP file http location where the recorded WAV file is posted.
That is from the jRecorder documentation(http://www.sajithmr.me/jrecorder/index.html), it seems that the file is not saved locally and is sent trough a post request to the mentioned php page on the host settings.
Add this in the jRecorder settings:
'host': 'acceptfile.php?filename=hello.wav'
And change the acceptfile.php to your php script that will handle the posted file.
Example php script for handeling the wav file(also from the documentation):
$upload_path = dirname(__FILE__). '/';
//here assume that filename parameter is passed. or your can write $filename= 'test.wav';
$filename = $_REQUEST['filename'];
$fp = fopen($upload_path."/".$filename.".wav", "wb");
fwrite($fp, file_get_contents('php://input'));
fclose($fp);
exit('done');
This script will save the audio file(wav) in the script folder.
I want to write a program that automatically downloads all the csv files on a webpage and then enter the data in those csv files into SQL Server. I have written the macro to enter the data from csv to SQL server. I just want you guys to help me in automatically downloading the files from a website everyday. is it similar to webscraping? I am new to web programming. So please guide me which languages to go through to do this
Quite easy with PowerShell:
$web = New-Object System.Net.WebClient
$web.Downloadstring("http://<your url here>") | out-file $env:tmp\MyFile.csv
Then use
Import-CSV
and the SQLServer PowerShell provider to inject your data into SQLServer.
I recommend Python, as usual.
After you figure out how to work with it, here are the modules I would use:
csv - for parsing CSV files.
urllib2 - for downloading files.
SQLAlchemy for database operations.
I also recommend PowerShell way to do the job.
Download data from web by WebClient like #David Brabant said
Then Use Import-Csv to get data row by row Import-Csv -Path file.csv
In the end, make use of SQL Server PowerShell Module in your script to upload data into SQL Server Invoke-Sqlcmd -Query '<sql statements>'
Here exists a complete sample script for you to download and learn. How to use SQL Server PowerShell Module to import data from CSV file