I have this PowerShell script:
Import-Module ActiveDirectory
Get-ADUser -filter 'SN -eq "Buyl"' -Properties UserPrincipalName,EmployeeNumber,Description,Department,GivenName,sn |
select #{N='UPN';E={$_.UserPrincipalName}},#{N='POINTER';E={$_.EmployeeNumber}},#{N='DESCRIPTION';E={$_.Description}},#{N='DEPARTMENT';E={$_.Department}},#{N='LASTNAME';E={$_.GivenName}},#{N='FIRSTNAME';E={$_.sn}},#{N='OU';E={$ou}} |
ConvertTo-Json
This works fine in PowerShell with users like Anaïs in their first name. I run this script from a PHP script that parses the Json output using json_decode. It seems the special characters break this function.
How can I make sure the PowerShell sends the output in clean UTF8 format that can be parsed with PHP?
Related
Trying to find out if there is a way to convert JSON to CSV locally on my computer without installing any 3rd party tools or using the browser/website. Maybe there is a way to get it done in command prompt or PowerShell or using notepad++ or SQL Server. I have tried Microsoft Excel's get data from JSON function but it errors out because it didn't like some characters in the JSON file. Thank you.
As Jeroen points out, ConvertFrom-Json, ConvertTo-Csv and Export-Csv will most likely be the basis for your solution. Without knowing the structure of your JSON, which is the key factor in determining how you'll go about it, here's an example with some native OS data on Windows using PowerShell.
Get-Process | Select-Object -Property Name,Handles,VM,Description -First 3 | ConvertTo-Json | Out-File .\procs.json
$procs = Get-Content -Path .\procs.json | ConvertFrom-Json
$procs | ConvertTo-Csv -Delimiter "`t" -NoTypeInformation | Out-File .\procs.csv
& .\procs.csv
In the code above, we...
Get an array of Windows process objects and create a JSON version of the data.
Convert the JSON data into an array of objects.
Convert the objects into a CSV.
Open the CSV with the default application, Excel in my case.
You should see a nicely formatted CSV with column headers and rows of data.
The txt file is just a bunch of UNC paths, i am trying to get a list of UNC paths from this text file put into another text file after the test-path is validated. it shows the validated paths on screen but the text file does not populate.
$cfgs = Get-Content .\cfgpath.txt
$cfgs | % {
if (Test-Path $_) { write-host "$_" | Out-File -FilePath c:\temp\1.txt -Append }
}
To complement Zam's helpful answer with background information:
Write-Host writes to the host[1] (typically, the console aka terminal), which bypasses PowerShell's success output stream and therefore sends nothing trough the pipeline.
See the bottom section of this answer for when Write-Host is appropriate; in short: you should generally only use it for display-only output.
Write-Output is the appropriate cmdlet for producing data output, but it is rarely necessary, because you can rely on PowerShell's convenient implicit output behavior, as shown in Zam's answer and explained in this answer.
Also, your command will perform much better if you simply pipe the % (ForEach-Object) command's output as a whole to a single Out-File call, rather than calling Out-File -Append for each input path.
Instead of using % with conditional explicit output, you can more elegantly implement your command with the Where-Object cmdlet:
Get-Content .\cfgpath.txt |
Where-Object { Test-Path $_ } |
Out-File -FilePath c:\temp\1.txt
Also note that for saving strings to a file it is more efficient to use Set-Content instead of
Out-File, though note that in Windows PowerShell the default output character encoding differs (no longer a concern in PowerShell [Core] 6+, which consistently defaults to BOM-less UTF-8); see this answer for when to choose which cmdlet.
By contrast, Out-File and > (its effective alias) use PowerShell's formatting system to write for-display representations of any non-string input objects to the output file, the same way that output renders to the display by default.
In other words: To save objects to a file in a way that is suitable for later programmatic processing, you need to use a structured file format, such as CSV (Export-Csv) or JSON (ConvertTo-Json, combined with Set-Content).
[1] In PowerShell 5.0 and higher, Write-Host now writes to a new stream, the information stream (number 6), which by default prints to the host. See about_Redirection.
Therefore, a 6> redirection now technically does allow you to send Write-Host output through the pipeline (though doing so is not a good idea) or capture / redirect it; e.g.,
Write-Host hi 6>&1 | % { "[$_]" }. Note that the type of the objects output by this redirection is System.Management.Automation.InformationRecord.
Write-Host only writes to the console. I believe what you want there is Write-Output.
$cfgs = Get-Content .\cfgpath.txt
$cfgs | % {
if (Test-Path $_) { write-output "$_" | Out-File -FilePath c:\temp\1.txt -Append }
}
Additionally you can just omit the Write-Output and that works too.
$cfgs = Get-Content .\cfgpath.txt
$cfgs | % {
if (Test-Path $_) { "$_" | Out-File -FilePath c:\temp\1.txt -Append }
}
I want to get a JSON object from the Google Direction API. The Google API response is formatted JSON with newlines like this:
{
geocoded_waypoints: [
{
geocoder_status: "OK",
place_id: "ChIJpws0Ra-DnkcRHLPUqU7qpiM",
types: [
"route"
]
}
]
}
But I need the response in one single line, without formatting.
Can I do this with curl? I downloaded the newest version.
I try this yet:
this version put each line in a row in the database
COPY "routing_import_gm" ("request") FROM PROGRAM 'curl "https://maps.googleapis.com/maps/api/directions/json?origin=48.xxxxxx,+11.yyyyyy&destination=48.xxxxxx,+11.yyyyyyy&key=***"';
cause erros:
COPY "routing_import_gm" ("request") FROM PROGRAM 'curl "https://maps.googleapis.com/maps/api/directions/json?origin=48.xxxxxx,+11.yyyyyy&destination=48.xxxxxx,+11.yyyyyyy&key=***" | tr -d ''\n'' ';
The following PowerShell command loads JSON data in any format from a given URL and outputs a single line of JSON as its result:
Invoke-RestMethod "https://your/url" | ConvertTo-Json -Compress -Depth 10
Invoke-RestMethod expects a JSON response, and decodes it into a data structure.
ConvertTo-Json takes a data structure and turns it into JSON.
-Compress skips all indentation and all newlines,
-Depth sets the maximum nesting level (which is only 2 by default, not enough for your data).
You can run PowerShell with an in-line command like this:
powershell -NoLogo -NonInteractive -Command "..."
And you could use that in your COPY statement.
COPY "routing_import_gm" ("request")
FROM PROGRAM 'powershell -NoLogo -NonInteractive -Command "Invoke-RestMethod "https://your/url" | ConvertTo-Json -Compress -Depth 10"';
You say you have trouble with Unicode characters. That is a rather tricky problem in this setup. There are two issues:
It requires proper configuration in multiple places to get Unicode characters across for shell commands. It's technically possible, but setting this up is a major pain.
JSON offers an ASCII-compatible format, which encodes all Unicode characters as \u..... That would side-step the issue, but ConvertTo-JSON in PowerShell 5.1 (which you have) does not offer that natively. The -EscapeHandling parameter (which does that) requires PowerShell 6.2.
We rely on ConvertTo-JSON to format the incoming JSON as a single line, and we can't simply change your PowerShell version. A possible way out would be to replace all non-ASCII characters with \u.... character codes using regex:
[regex]::Replace($json, '[^\x00-\x7f]', {param($m) '\u{0:X4}'-f[int][char]$m[0].value})
If $json is '{"test": "Smørrebrød 🥪"}', the above returns '{"test": "Sm\u00F8rrebr\u00F8d \uD83E\uDD6A"}', which is both valid JSON and ASCII-only, so it won't give you any trouble with COPY FROM PROGRAM.
Applied to the code above, we would end up with this:
powershell -NoLogo -Command "$json = Invoke-RestMethod 'https://your/url' | ConvertTo-Json -Compress -Depth 10; [regex]::Replace($json, '[^\x00-\x7f]', {param($m) '\u{0:X4}'-f[int][char]$m[0].value})"
Notes
While all of this is pretty convoluted, the advantage of using PowerShell is
All of it (requesting the URL, getting the JSON from the response, converting it) is Unicode-aware and never crosses process boundaries, as e.g. piping from curl for Windows into jq or sed or tr for Windows would. Crossing process boundaries always has a chance of mangling your Unicode data.
If the server itself returns literal Unicode characters in the JSON, they would get converted to \u.... as well.
PowerShell is readily available on every Windows machine, other tools must be installed first.
If you can upgrade your PowerShell version, you could drop the regex part and use ConvertTo-JSON -EscapeHandling EscapeNonAscii instead.
I am trying to generate an html page with file index. This approach worked seamlessly:
$htmlout = Get-ChildItem -Path "$SearchPath" -Filter "$fileType" -Recurse |
Select #{Name="Link";Expression={("<a rel=" + $_.FullName + " href=file:///" + $_.FullName + ">$_</a>")}}
The Link column had file names only (i.e. test.txt) and displayed file content when clicking on it. Then we've got an additional requirement to skip old files. The script is now:
$htmlout = Get-ChildItem -Path "$SearchPath" -Recurse -include ("$fileType") | Where-Object {$_.LastWriteTime -ge "01/01/2014"} |` Select #{Name="Link";Expression={("<a rel=" + $_.FullName + " href=file:///" + $_.FullName + ">$_</a>")}}
It still works, but Link column now displays the entire file path + file name (i.e. \fileserver\folder1\folder2\test.txt).
Adding >$_.Name< does not work here.
I am trying to understand why the same URL line behaves differently after filter change.
Background
As far as I can tell, there is a discrepancy with the interaction between the .ToString() method and the DefaultDisplayProperty of objects returned by Get-ChildItem.
The behavior manifests when both of the following conditions are true:
- The -filter parameter is being used.
- The value of the -Path parameter resolves to a single directory, whether or not -Recurse is used.
Under the above circumstances, the .ToString() method implemented by PowerShell uses the Name property as default, rather than FullName as is the case in all other scenarios.
My guess is that this inconsistency is due to the underlying object types returned by the FileSystem provider when -Filter is used, rather than the objects PowerShell returns when it handles the search/filter itself (as is the case with -Include).
Observation
When you wrap your $_ pipeline object variable in double quotes, PowerShell's type-conversion implicitly calls the .ToString() method and you get the resulting name variation.
Solution
To correct your issue, you could simply use -Filter in both code examples and get the desired output, however, that is prone to cause problems sooner or later.
The more appropriate way to negate the problem is to properly use a PowerShell sub-expression within the double-quoted strings.
To create a sub-expression, simply wrap the desired code like so: $(). This creates a separation between which characters are code and which are part of the string; in your case allowing you to use the member access operator .. The method also alleviates the need to do string concatenation with the + operator.
Solution Code:
$HTMLOut = Get-ChildItem -Path $SearchPath -Recurse -Include $FileType | Where-Object {$_.LastWriteTime -ge "01/01/2014"} | Select #{Name="Link";Expression={("<a rel=$($_.FullName) href=file:///$($_.FullName)>$($_.Name)</a>")}}
I have been given a working PowerShell script to modify. This script checks for broken links in a SharePoint web application and reports any broken links it finds. Currently, the script outputs the results to a text file.
Code:
$results | Out-File report.txt
Result Format:
Name of link list - ID of the item - URL + URL Name - HTTP status code
- URL
Result:
Link List 1 - 1 - http://google.com, Google (Good) - 200 -
http://google.com
However, when I try to use the ConvertTo-HTML function I get a different output:
Code:
$results | ConvertTo-Html | Out-File report.html
Result:
76
Therefore, instead of receiving a string of text I am receiving its length.
What am I doing wrong here?
Note:
results is an array.
Convertto-Html is not meant to be used like that, from help:
Converts Microsoft .NET Framework objects into HTML that can be displayed in a Web browser.
You need to send an object (resulting from a command) to this cmdlet, not just a bunch of text. See: Get-Help Convertto-Html -Examples
this is a way to do what you want:
$results | ForEach-Object {Add-Member -InputObject $_ -Type NoteProperty -Name Value -Value $_; $_} | ConvertTo-Html -Property Value