When working on large SSIS projects containing several packages, the packages can start to get a bit messy with variables that were created but never used or have been made redundant by other changes. I need something that will scan several SSIS packages and list all unused variables.
I have managed to answer my own question by employing some Powershell. The script below uses xpath to get the variable names and then uses regex to find the number of occurrences, if it occurs once it must be because it was defined but never used.
The only caveat is that if you use names for variables that are words that would naturally be present in a dtsx file, then the script will not pick them up. I probably need to expand my script to only do a regex search on spesific nodes in the package.
$results = #()
Get-ChildItem -Filter *.dtsx |
% {
$xml = [xml](Get-Content $_)
$Package = $_.Name
$ns = [System.Xml.XmlNamespaceManager]($xml.NameTable)
$ns.AddNamespace("DTS", "www.microsoft.com/SqlServer/Dts")
$var_list = #($xml.SelectNodes("//DTS:Variable/DTS:Property[#DTS:Name = 'ObjectName']", $ns) | % {$_.'#text'})
$var_list | ? {#([Regex]::Matches($xml.InnerXml, "\b$($_)\b")).Count -eq 1} |
% { $results += New-Object PSObject -Property #{
Package = $Package
Name = $_}
}
}
$results
This is a great question, as I have the same concern with a few rather large SSIS packages. Unfortunately, external to the SSIS packages, there isn't any feature available that will provide this function. See discussions in attached link:
CodePlex
But by opening an SSIS package, you can determine the variable usage by applying the steps outlined in the following link:
find-ssis-variable-dependencies-with-bi-xpress
Hope this helps.
Related
I found this script on this site <thanks Nick!>
$files = gci $srcpath
foreach ($srcfile in $files) {
# Build destination file path
$dstfile = [string]($dstpath, '\', $srcfile.name -join '')
# Copy the file
cp $srcfile.FullName $dstfile.FullName -whatif
# Make sure file was copied and exists before copying over properties/attributes
if ($dstfile.Exists) {
# $dstfile.CreationTime = $srcfile.CreationTime
# $dstfile.LastAccessTime = $srcfile.LastAccessTime
$dstfile.LastWriteTime = $srcfile.LastWriteTime
# $dstfile.Attributes = $srcfile.Attributes
# $dstfile.SetAccessControl($srcfile.GetAccessControl())
}
}
I want to turn this into a function so it accepts recursive directories to copy the timestamps from the source, a cloud folder to it's equivalent at the destination.
Now, I have tried calling the function using multiple variables, trying different methods such as:
$src = $args[0]
$dst = $args[1]
Get-Timestamp $src, $dst
either uses the default folder that the script is running or will fail when it tries to list the contents combining the 2 variables together.
even setting up the function like so
[CmdletBinding(DefaultParamerterSetName='Srcpath')]
param (
[Parameter(Mandatory = $true,
ParameterSetName = 'Srcpath',
Position = 0)]
[string[]]$Srcpath,
# [Parameter(Mandatory = $true,
# ParameterSetName = 'DSTpath',
# Position = 1)]
[string]$DSTpath
)
$PSCmdlet.ParameterSetName
is not producing the expected result.
This will work on its own, doing one folder at a time. But not when doing subfolders.
Suggestions would be greatly appreciated.
From the PowerShell ISE or VSCode select a simple or advanced function snippet.
PowerShell ISE - use CRTL+J, type function, hit enter and you get this:
function MyFunction ($param1, $param2)
{
}
VSCode - use CRTL=ALT+J, type function, hit enter and you get this:
function FunctionName {
param (
OptionalParameters
)
}
Put your code below the param block. Now, both those function names are not best practice and should be verb-noun, as documented here:
About Functions | MSDocs
Function Names
You can assign any name to a function, but functions
that you share with others should follow the naming rules that have
been established for all PowerShell commands.
Functions names should consist of a verb-noun pair in which the verb
identifies the action that the function performs and the noun
identifies the item on which the cmdlet performs its action.
Functions should use the standard verbs that have been approved for
all PowerShell commands. These verbs help us to keep our command names
simple, consistent, and easy for users to understand.
For more information about the standard PowerShell verbs, see Approved
Verbs in the Microsoft Docs.
You are not getting a recursive search because you are not telling it to. That is what the -Recurse of the Get-ChildItem cmdlet is for.
# Get specifics for a module, cmdlet, or function
(Get-Command -Name Get-ChildItem).Parameters
(Get-Command -Name Get-ChildItem).Parameters.Keys
# Results
<#
Path
LiteralPath
Filter
Include
Exclude
Recurse
Depth
Force
Name
Verbose
Debug
ErrorAction
WarningAction
InformationAction
ErrorVariable
WarningVariable
InformationVariable
OutVariable
OutBuffer
PipelineVariable
UseTransaction
Attributes
Directory
File
Hidden
ReadOnly
System
#>
Get-help -Name Get-ChildItem -Examples
# Results
<#
Example 3: Get child items in the current directory and subdirectories
Get-ChildItem -Path C:\Test\*.txt -Recurse -Force
Directory: C:\Test\Logs\Adirectory
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 2/12/2019 16:16 20 Afile4.txt
-a-h-- 2/12/2019 15:52 22 hiddenfile.txt
-a---- 2/13/2019 13:26 20 LogFile4.txt
Directory: C:\Test\Logs\Backup
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 2/12/2019 16:16 20 ATextFile.txt
-a---- 2/12/2019 15:50 20 LogFile3.txt
#>
Get-help -Name Get-ChildItem -Full
Get-help -Name Get-ChildItem -Online
Lastly, aliases/shorthand names are for interactive throw-away code, not in production/shared scripts. As discussed here:
• Best Practices for aliases
https://devblogs.microsoft.com/scripting/best-practice-for-using-aliases-in-powershell-scripts
https://devblogs.microsoft.com/scripting/using-powershell-aliases-best-practices
Why worry about aliases in the first place? ... There are two things
at work when it comes to a script. The first is that no alias is
guaranteed to exist —even aliases that are created by Windows
PowerShell. ...
... and if you create custom ones, you can step on one already configured on a host.
Lastly, PSScriptAnalyzer, VSCode, etc., make all known aliases as errors, until you make them their full name. Custom aliases, you need to expand yourself. Never assume folks know them or care to. PowerShell is verbose for a reason (good/bad/indifferent is all opinion), easy to read for the most inexperienced, self-documenting, and easier to maintain. Make sure to read the available PowerShell Best Practice references that are available and remember, you write code for those who will use it or follow you or maintain it.
The code below is part of a switch and it's working fine, but the problem is: I need to change my file name to 15... Is it possible to to change it so that when I start it, it waits to select for a file with the tab key? Something like when you write Import-Csv in a PowerShell console and press Tab it shows all possbile paths and files.
$names = Import-Csv 15.csv -Header Givenname,Surname -Delimiter ";"
Write-Host "Rename your csv file to '15' and put it in same folder with this script" -ForegroundColor Cyan
pause
foreach ($Name in $Names) {
$FirstFilter = $Name.Givenname
$SecondFilter = $Name.Surname
Get-ADUser -Filter {GivenName -like $FirstFilter -and Surname -like $SecondFilter} |
select Enabled, SamAccountName, DistinguishedName,
#{n="ou";e={($_.DistinguishedName -split ",*..=")[2]}} |
Export-Csv .\sam.csv -NoTypeInformation -Append
}
So you want Intellisense in your script. Ambitious move. Most people would settle for the file browser dialog box. Anyway, I am going to have to refer you to smarter men than me. I was thinking ValidateSet attribute would serve your purpose but I realized that the traditional param block is not enough. So I looked up DynamicParams and this is what I found. This should work for you.
https://blogs.technet.microsoft.com/pstips/2014/06/09/dynamic-validateset-in-a-dynamic-parameter/
The simplest solution is to make your script accept the target file as an argument, by declaring a parameter:
param(
# Declare a mandatory parameter to which the file path of the CSV
# file to import must be passed as an argument on invocation.
[Parameter(Mandatory)]
[string] $FilePath
)
$names = Import-Csv $FilePath -Header Givenname,Surname -Delimiter ";"
foreach ($Name in $Names) {
$FirstFilter = $Name.Givenname
$SecondFilter = $Name.Surname
Get-ADUser -Filter {GivenName -like $FirstFilter -and Surname -like $SecondFilter} |
select Enabled, SamAccountName, DistinguishedName,
#{n="ou";e={($_.DistinguishedName -split ",*..=")[2]}} |
Export-Csv .\sam.csv -NoTypeInformation -Append
}
If you invoke your script without a file path, you will be prompted for it; let's assume your script is located in the current dir. and its name is someScript.ps1:
./someScript # invocation with no argument prompts for a value for $FilePath
Unfortunately, such an automatic prompt is not user-friendly and offers no tab completion.
However, on the command line PowerShell's tab completion defaults to completing file and directory names in the current location, so that:
./someScript <press tab here>
cycles through all files and directories in the current folder.
You can even type a wildcard expression and tab-complete that, if you don't know the full filename or don't want to type it in full:
./someScript *.csv<press tab here>
This will cycle through all *.csv files in the current dir. only.
If you want to go even further and customize tab completion to only cycle through *.csv files, you can use an [ArgumentCompleter({ ... })] attribute (PSv5+):
param(
[Parameter(Mandatory)]
# Implement custom tab-completion based on only the *.csv files in the current dir.
[ArgumentCompleter({
param($cmd, $param, $wordToComplete)
Get-ChildItem -Name "$wordToComplete*.csv"
})]
[string] $FilePath
)
# ...
Now,
./someScript <tab>
will cycle only through the *.csv files in the current directory, if any.
Caveat: As of PowerShell 7.0, tab-completing an argument for which the ArgumentCompleter script block returns no matches (in this case, with no *.csv files present) unexpectedly falls back to the default file- and directory-name completion - see this GitHub issue.
Similarly,
./someScript 1<tab>
will cycle only through the *.csv files in the current directory whose name starts with 1, if any.
As an alternative to using an attribute as part of a script's / function's definition, you can use the PSv5+ Register-ArgumentCompleter cmdlet to attach tab completions to the parameters of any command, i.e., including preexisting ones.
In PSv4- you have two (cumbersome) options for custom tab completion:
Use a dynamic parameter with a dynamically constructed [ValidateSet()] attribute - see the link in Rohin Sidharth's answer.
Customize the tabexpansion2 (PSv3, PSv4) / tabexpansion (PSv1, PSv2) function, but be sure not to accidentally replace existing functionality.
Below is my example.ps1 file that I use to write 1-off scripts. In your case I think you can get what you want with it. For example (no pun intended) you could call this script by typing
C:\PathToYourScripts\example.ps1 [tab]
where [tab] represents pressing the tab key. Powershell intellisense will kick in and offer autocompletion for file names. If your .csv file is not in the current director you can easily use Powershell intellisense to help you find it
C:\PathToYourScripts\example.ps1 C:\PathToCSvFiles[tab]
and powershell will autocomplete. Would-be-downvoters might notice that powershell autocomplete is definitely NOT a complete file-picker but this seems to fulfill the intent of the asked question. Here's the sample.
<#
.NOTES
this is an example script
.SYNOPSIS
this is an example script
.DESCRIPTION
this is an example script
.Example
this is an example script
.LINK
https://my/_git/GitDrive
#>
[CmdletBinding(SupportsShouldProcess=$True, ConfirmImpact="Low")]
param (
[string] $fileName
)
Begin {
}
Process {
if ($PSCmdlet.ShouldProcess("Simulated execution to process $($fileName): Omit -Whatif to process ")) {
Write-Information -Message "Processing $fileName" -InformationAction Continue
}
}
End {
}
If you want to get autocomplete help for multiple parameters just type in the parameter name(s) and press [tab] after each one. Note that leaving the parameters blank will not break the script but you can either extend this to mark the parameters required or just fail with a helpful message. That seems a bit beyond the original question so I'll stop here.
I've seen different post regarding paralleling or run functions simultaneously in parallel but the code in the answers have not quite worked for me. I'm doing patching automation and the functions I have all work and do their thing separately but since we work with more than 200+ computers, waiting for each function to finish with its batch of computers kind of defeats the purpose. I have the code in one script and in summary its structured like this:
define global variables
$global:varN
define sub-functions
function sub-function1()
function sub-functionN()
define main functions
function InstallGoogleFunction($global:varN)
{
$var1
$result1 = sub-function1
$resultN = sub-functionN
}
function InstallVLCFunction($global:varN)
{
"similar code as above"
}
function InstallAppFunction($global:varN)
{
"similar code as above"
}
The functions will all install a different app/software and will write output to a file. The only thing is I cannot seem to run all the functions for installation without waiting for the first one to finish. I I tried start-job code but it executed and displayed a table like output but when verifying the computers neither had anything running on Task Manager. Is there a way powershell can run this installation functions at the same time? If I have to resort to a one-by-one I will call the functions by the least amount of time taken or the least computers the functions read they need to install I will but I just wanted someone to better explain if this can be done.
You can use multithreading to achieve this. Below example shows how you can trigger tasks on multiple using multi threading in PowerShell.
Replace list of machines in $Computers with your machines and give it a try. Below example get the disk details on given machines
# This is your script you want to execute
$wmidiskblock =
{
Param($ComputerName = "LocalHost")
Get-WmiObject -ComputerName $ComputerName -Class win32_logicaldisk | Where-Object {$_.drivetype -eq 3}
}
# List of servers
$Computers = #("Machine1", "Machine2", "Machine3")
#Start all jobs in parallel
ForEach($Computer in $Computers)
{
Write-Host $Computer
Start-Job -scriptblock $wmidiskblock -ArgumentList $Computer
}
Get-Job | Wait-Job
$out = Get-Job | Receive-Job
$out |export-csv 'c:\temp\wmi.csv'
I'm new to SharePoint, but I'm looking to export all choices in a Choice field in a SharePoint list to a .csv file, using powershell.
Hopefully the file will look like:
"Choices"
"Choice1"
"Choice2"
"Choice3"
and so on. I need the choices in a .csv file, that I use for inputs in another powershell script.
I have tried googling, but every result seems to be on how to export the result for a specific choice, like the example below:
http://blog.metrostarsystems.com/2015/06/05/using-powershell-and-the-sharepoint-2013-csom-to-export-list-data/
Context:
We maintain a list of IPs (and relevant information) that has access to our test websites. We maintain the list and then manually white-list the IPs on the websites. Since we have 20 websites at the moment and the number is increasing, we want to automate it, so we only have to maintain the SharePoint list and "magic" white-lists the IPs on the websites. The choice field mentioned is the websites, hence when we add new websites, I don't want to update a .txt or csv file manually, I just want it to be created based on the choices available in the SharePoint list.
If you're running script on a SharePoint web front end server, use the server object model
If you're not running this from the SharePoint management shell, you'll want to start by adding the SharePoint snapin
add-pssnapin "Microsoft.SharePoint.Powershell" -ErrorAction SilentlyContinue
Now that you have access to the SharePoint object model, you can get the list...
$web = get-spweb http://your.web.url
$list = $web.lists["List Title"]
Then get the field...
$field = $list.Fields | where-object {$_.title -eq "Field Title"}
# you can also get the field by its internal name like so:
# $field = $list.Fields.GetFieldByInternalName("FieldInternalName")
Then export the field choices to a CSV and dispose of the SPWeb object.
$field.choices | Export-Csv -path "\WhereverYouWant\YourFile.csv" -notype
$web.dispose()
If you're running the script from outside the SharePoint environment, use the client object model
These approach requires a bit of C# code due to Powershell not playing so nice with .NET methods that expect generic types.
The gist of it is that you'll write a static C# method that you can invoke from Powershell to get the data you want. In this case, getting the field schema as XML is a great way to get at the available field choices.
# Replace these paths with actual, locally accessible paths to the Client and Client.Runtime DLLs
$hive = "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14"
$assemblies = #()
$assemblies += $hive+"\ISAPI\Microsoft.SharePoint.Client.dll"
$assemblies += $hive + "\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"
$assemblies += "System.Core"
# Code for accessing the SharePoint client object model
$cSharp = #"
using System;
using System.Collections.Generic;
using Microsoft.SharePoint.Client;
namespace SPClient
{
public class CustomClass{
public static string GetFieldSchema()
{
ClientContext clientContext = new ClientContext(`"http://your/SharePoint/url`");
List list = clientContext.Web.Lists.GetByTitle(`"List Name Here`");
Field field = list.Fields.GetByInternalNameOrTitle(`"Field Name Here`");
clientContext.Load(field);
clientContext.ExecuteQuery();
return field.SchemaXml;
}
}
}
"#
# Here's the magic where you load the above code and reference the assemblies
Add-Type -TypeDefinition $cSharp -ReferencedAssemblies $assemblies
# Now you can invoke the custom code to get the XML string
[xml]$schema = [SPClient.CustomClass]::GetFieldSchema()
# Parse the XML as normal
$array = #()
$schema.Field.Choices.Choice | %{ $array += new-object psobject -property #{Choice=$_} }
$array | Export-CSV -notype -path "\WhereverYouWant\YourFile.csv"
Retrieving the site object then referencing the .choices property does the trick.
For a client operating on a 2007 environment code is as follows.
$spsite=[Microsoft.SharePoint.SPSite]("<site url>")
$rootWebSite=$spsite.RootWeb
$rootWebSite.Fields[0].Choices
I've put my functions in a separate file and I call the file with:
$workingdir = Split-Path $MyInvocation.MyCommand.Path -Parent
. "$workingdir\serverscan-functions.ps1"
But, if I call the scripts like
my-function
how will the variable scope (from within "my-function") be?
Should I still $script:variable to make the variable exist outside the function or have I dot-sourced the function as well?
Hope I don't confuse anyone with my question... I've tried to make it as understandable as possible, but still learning all the basic concept so I find it hard to explain..
When you dot source code it will behave as if that code was still in the original script. The scopes will be the same as if it was all in one file.
C:\functions.ps1 code:
$myVariable = "Test"
function Test-DotSource {
$script:thisIsAvailableInFunctions = "foo"
$thisIsAvailableOnlyInThisFunction = "bar"
}
main.ps1 code
$script:thisIsAvailableInFunctions = ""
. C:\functions.ps1
# Call the function to set values.
Test-DotSource
$script:thisIsAvailableInFunctions -eq "foo"
# Outputs True because of the script: scope modifier
$thisIsAvailableOnlyInThisFunction -eq "bar"
# Outputs False because it's undefined in this scope.
$myVariable -eq "Test"
# Outputs true because it's in the same scope due to dot sourcing.
In order to achieve what you want, you'll probably need to create a module. In the module, export the functions using Export-ModuleMember, and as long as you don't explicitly export any variables as module members, you should be fine.
Once you've created the module, import it using the Import-Module cmdlet.
My 2cents:
Usually (after a past Andy Arismendi answer! God bless you man!) I save all my scripts in $pwd folder (added in system path environment). The I can call them from the console wihtout dot sourcing and no script variable poisoning the console after a script ends his job.
If you cannot modify yours functions in simple scripts (sometimes it happens) I'm agree with Trevor answer to create a module and import it in $profile