Is there a way to connect to a network drive that requires a different username/password than the username/password of the user running the package?
I need to copy files from a remote server. Right now I map the network drive in Windows Explorer then do I filesystem task. However, eventually this package will be ran automatically, from a different machine, and will need to map the network drive on its own. Is this possible?
You can use the Execute Process task with the "net use" command to create the mapped drive. Here's how the properties of the task should be set:
Executable: net
Arguments: use \Server\SomeShare YourPassword /user:Domain\YourUser
Any File System tasks following the Execute Process will be able to access the files.
Alternative Method
This Sql Server Select Article covers the steps in details but the basics are:
1) Create a "Execute Process Task" to map the network drive (this maps to the z:)
Executable: cmd.exe
Arguments: /c "NET USE Z: "\\servername\shareddrivename" /user:mydomain\myusername mypassword"
2) Then run a "File System Task" to perform the copy. Remember that the destination "Flat File Connection" must have "DelayValidation" set to True as z:\suchandsuch.csv won't exist at design time.
3) Finally, unmap the drive when you're done with another "Execute Process Task"
Executable: cmd.exe
Arguments: /c "NET USE Z: /delete"
Why not use an FTP task to GET the files over to the local machine? Run SSIS on the local machine. When transferring using FTP in binary, its real fast. Just remember that the ROW delimter for SSIS should be LF, not CRLF, as binary FTp does not convert LF (unix) to CRLF (windows)
You have to map the network drive, here's an example that I'm using now:
profile = "false"
landingPadDir = Dts.Variables("strLandingPadDir").Value.ToString
resultsDir = Dts.Variables("strResultsDir").Value.ToString
user = Dts.Variables("strUserName").Value.ToString
pass = Dts.Variables("strPassword").Value.ToString
driveLetter = Dts.Variables("strDriveLetter").Value.ToString
objNetwork = CreateObject("WScript.Network")
CheckDrive = objNetwork.EnumNetworkDrives()
If CheckDrive.Count > 0 Then
For intcount = 0 To CheckDrive.Count - 1 Step 2 'if drive is already mapped, then disconnect it
If CheckDrive.Item(intcount) = driveLetter Then
objNetwork.RemoveNetworkDrive(driveLetter)
End If
Next
End If
objNetwork.MapNetworkDrive(driveLetter, landingPadDir, profile, user, pass)
From There just use that driveLetter and access the file via the mapped drive.
I'm having one issue (which led me here) with a new script that accesses two share drives and performs some copy/move operations between the drives and I get an error from SSIS that says:
This network connection has files open or requests pending.
at Microsoft.VisualBasic.CompilerServices.LateBinding.InternalLateCall(Object o, Type objType, String name, Object[] args, String[] paramnames, Boolean[] CopyBack, Boolean IgnoreReturn)
at Microsoft.VisualBasic.CompilerServices.NewLateBinding.LateCall(Object Instance, Type Type, String MemberName, Object[] Arguments, String[] ArgumentNames, Type[] TypeArguments, Boolean[] CopyBack, Boolean IgnoreReturn)
at ScriptTask_3c0c366598174ec2b6a217c43470f581.ScriptMain.Main()
This is only on the "2nd run" of the process and if I run it a 3rd time it all works fine so I'm guessing the connection isn't being properly closed or it is not waiting for the copy/move to complete before moving forward or some such, but I'm unable to find a "close" or "flush" command that prevents this error. If you have any solution, please let me know, but the above code should work for getting the drive mapped using your alternate credentials and allow you to access that share.
Zach
To make the package more robust, you can do the following;
In the first Execute Process Task, set - FailTaskIfReturnCodeNotSuccessValue = False
This will let the package run if the last disconnect has not worked.
This is an older question but more recent versions of SQL Server with SSIS databases allow you to use a proxy to execute SQ Server jobs.
In SSMS Under Security<Credentials set up a credential in the database mapped to the AD account you want to use.
Under SQL Server Agent create a new proxy giving it the credential from step 1 and permissions to execute SSIS packages.
Under the SQL Server Agent jobs create a new job that executes your package
Select the step that executes the package and click EDIT. In the Run As dropdown select the Proxy you created in step 2
Related
I would like to use test databases for feature branches.
Of course it would be best to create a gitlab ci environment on the fly (review apps style) and also create a test database on the target system with the same name. Unfortunately, this is not possible because the MySQL databases in the target system have fixed names, like xxx_1, xxx_2 etc. and this cannot be changed without moving to a different hosting provider.
So I would like to do something like "grab an empty test data base from the given xxx_n and then empty it again when the branch is deleted".
How could this be handled with gitlab ci?
Can I set a variable on the project that says "feature branch Y already uses database xxx_4"?
Or should I put a table into the test database to store this information?
Using dynamic environments/variables and stop jobs might be able to do the trick. Stop jobs will run when the environment is "stopped" -- in the case of feature branches without associated MRs, when the feature branch is deleted (or if there is an open MR for the review app, when the MR is merged or closed)
Can I set a variable on the project that says "feature branch Y already uses database xxx_4"?
One way may be to put the db name directly in the environment name. Then the Environments API keeps track of this.
stages:
- pre-deploy
- deploy
determine_database:
stage: pre-deploy
image: python:3.9-slim
script:
- pip install python-gitlab
- database_name=$(determine-database) # determine what database names are not currently in use
- echo "database_name=${database_name}" > vars.env
artifacts:
reports: # automatically set $database_name variable in subsequent jobs
dotenv: "vars.env"
deploy_review_app:
stage: deploy
environment:
name: review/$CI_COMMIT_REF_SLUG/$database_name
on_stop: teardown
script:
- echo "deploying review app for $CI_COMMIT_REF with database name configuration $database_name"
- ... # steps to actually do the deploy
teardown: # this will trigger when the environment is stopped
stage: deploy
variables:
GIT_STRATEGY: none # ensures this works even if the branch is deleted
when: manual
script:
- echo "tearing down test database $database_name"
- ... # actual script steps to stop env and cleanup database
environment:
name: review/$CI_COMMIT_REF_SLUG/$database_name
action: "stop"
The implementation of the determine-database command may have to connect to your database to determine what database names are available (or perhaps you have a set of these provisioned in advance). You can then inspect the GitLab environments API to see what database names are still in use (since it's baked into the environment name).
For example, you might have something like this. Here, I am using the python-gitlab API wrapper just because it's most familiar to me, but the same principle can be applied to any method of calling the GitLab REST API.
#!/usr/bin/env python3
import gitlab
import os, sys, random
GITLAB_URL = os.environ['CI_SERVER_URL']
PROJECT_TOKEN = os.environ['MY_PROJECT_TOKEN'] # you generate and add this to your CI/CD variables!
PROJECT_ID = os.environ['CI_PROJECT_ID']
DATABASE_NAMES = ['xxx_1', 'xxx_2', 'xxx_3'] # or determine this programmatically by connecting to the DB
gl = gitlab.Gitlab(GITLAB_URL, private_token=PROJECT_TOKEN)
in_use_databases = []
project = gl.projects.get(PROJECT_ID)
for environment in project.environments.list(state='available', all=True):
# the in-use database name is the string after the last '/' in the env name
in_use_db_name = environment.name.split('/')[-1]
in_use_databases.append(in_use_db_name)
available_databases = [name for name in DATABASE_NAMES if name not in in_use_databases]
if not available_databases: # bail if all databases are in use
print('FATAL. no available databases', file=sys.stderr)
raise SystemExit(1)
# otherwise pick one and output to stdout
db_name = random.choice(available_databses)
# optionally you could prepare the database here, too, instead of relying on the `on_stop` job.
print(db_name)
There is a potential concurrency problem here (two runs of determine_database concurrently on different branches can potentially select the same db twice before either finish) but that could be addressed with resource locks.
I have a SSIS package that will load data from a set of excel files then archive those files to a specified folder.
The excel files are stored in a folder and inside that folder I have the archive folder.
Below is my script task code for reference.
public void Main()
{
// TODO: Add your code here
string sourceDir = Dts.Variables["User::strFilePath"].Value.ToString();
string destDir = Dts.Variables["User::strArchivePath"].Value.ToString();
DirectoryInfo di = new DirectoryInfo(sourceDir);
string[] sDirFiles = Directory.GetFiles(sourceDir);
FileInfo[] fi = di.GetFiles("*.xls");
int fileCnt = fi.Length;
for (int i = 0; i < fileCnt; i++)
{
String filename = fi[i].Name;
string[] splitFilename = filename.Split('.');
DateTime dt = DateTime.Now;
string ArchiveDate = String.Format("{0:ddMMMyyyy}", dt);
string sourceFileName = filename;
string sourceFilePath = sourceDir + filename;
string destinationFileName = splitFilename[0] + '_' + ArchiveDate + '.' + splitFilename[1];
string destinationPath = destDir + destinationFileName;
//MessageBox.Show("Source File " + sourceFilePath + " to destination " + destinationPath);
if (File.Exists(destinationPath))
File.Delete(destinationPath);
// To move a file or folder to a new location:
System.IO.File.Move(sourceFilePath, destinationPath);
}
Dts.TaskResult = (int)ScriptResults.Success;
}
The sourceDir and destDir are variables that provides the path of source files folder and archive folder. The package works fine when I run it from visual studio.
I have deployed it to run as a job using deployment utility by creating manifest file. When I run the job I'm getting error in my archive script task. Below is the screenshot of it.
I searched for solution in stackoverflow however the provided solutions does not solve my problem.
Problem using SQL Agent to run SSIS Packages - fails with “DTSER_FAILURE(1)”
DTSX package runs in Visual Studio but not when called from a Database Job
I have granted read write access to SQL server for both the folders. Still getting the same error?
Any help is highly appreciated.
Try choosing 64bit on your Project --> Properties --> Dubugging--> Run64BitTime this could solve the problem.
I replaced the script task with a file system task and it worked. I also created a proxy account to run the package with my system credentials so that it will have the access to the folder specified.
Putting the steps in creating a proxy account for running the SQL job so that it will be helpful for users looking out for it..
Create a User account in SQL under which you need to run the Job
Create a credentials for the user account created.
--Script #1 - Creating a credential to be used by proxy
USE MASTER
GO
--Drop the credential if it is already existing
IF EXISTS (SELECT 1 FROM sys.credentials WHERE name = N'SSISProxyCredentials')
BEGIN
DROP CREDENTIAL [SSISProxyCredentials]
END
GO
CREATE CREDENTIAL [SSISProxyCredentials]
WITH IDENTITY = N'<Insert the Username>',
SECRET = N'abcd#0987'
GO
Create Proxy account and associate the credentials created
--Script #2 - Creating a proxy account
USE msdb
GO
--Drop the proxy if it is already existing
IF EXISTS (SELECT 1 FROM msdb.dbo.sysproxies WHERE name = N'SSISProxyDemo')
BEGIN
EXEC dbo.sp_delete_proxy
#proxy_name = N'SSISProxyDemo'
END
GO
--Create a proxy and use the same credential as created above
EXEC msdb.dbo.sp_add_proxy
#proxy_name = N'SSISProxyDemo',
#credential_name=N'SSISProxyCredentials',
#enabled=1
GO
--To enable or disable you can use this command
EXEC msdb.dbo.sp_update_proxy
#proxy_name = N'SSISProxyDemo',
#enabled = 1 --#enabled = 0
GO
Granting proxy account to SQL Server Agent Sub-systems
USE msdb
GO
--You can view all the sub systems of SQL Server Agent with this command
--You can notice for SSIS Subsystem id is 11
EXEC sp_enum_sqlagent_subsystems
GO
--Grant created proxy to SQL Agent subsystem
--You can grant created proxy to as many as available subsystems
EXEC msdb.dbo.sp_grant_proxy_to_subsystem
#proxy_name=N'SSISProxyDemo',
#subsystem_id=11 --subsystem 11 is for SSIS as you can see in the above image
GO
--View all the proxies granted to all the subsystems
EXEC dbo.sp_enum_proxy_for_subsystem
Granting proxy access to security principals
USE msdb
GO
--Grant proxy account access to security principals that could be
--either login name or fixed server role or msdb role
--Please note, Members of sysadmin server role are allowed to use any proxy
EXEC msdb.dbo.sp_grant_login_to_proxy
#proxy_name=N'SSISProxyDemo'
,#login_name=N'<Insert the Username>'
--,#fixed_server_role=N''
--,#msdb_role=N''
GO
--View logins provided access to proxies
EXEC dbo.sp_enum_login_for_proxy
GO
Finally associate the proxy account to the package step. This can be also done through job wizard.
EXEC msdb.dbo.sp_add_jobstep #job_id=#jobId, #step_name=N'SSISPackageCall',
#step_id=1,
#cmdexec_success_code=0,
#on_success_action=1,
#on_success_step_id=0,
#on_fail_action=2,
#on_fail_step_id=0,
#retry_attempts=0,
#retry_interval=0,
#os_run_priority=0, #subsystem=N'SSIS',
#command=N'/FILE "C:\Package.dtsx" /CHECKPOINTING OFF /REPORTING E',
#database_name=N'master',
#flags=0,
#proxy_name = N'SSISProxyDemo';
Thanks for the valuable replies for my post..
I'm using SSDT for Visual Studio 2012 here and using its command-line tool SQLPackage.exe to publish a .dacpac file.
I want to publish that to an SQL Server database project. I'm trying to use parameter at this guide but cannot find ways how to do it.
How can I do that?
Are you trying to publish to a Database, or create a database project from a dacpac? These are two different things.
To create a database project based on a dacpac, create a new SQL Server Database Project in Visual Studio. Then right-click on the project in Solution Explorer and choose "Import -> Data-tier Application (*.dacpac)" and select your dacpac. That will convert the contents of the dacpac into scripts in the project, and if you choose "Import database settings" the database options will be set based on the settings in the dacpac. Unfortunately there's no way to do this via a command line tool that I'm aware of.
To use SqlPackage.exe to publish to a database, the guide you linked to has all the options. A simple command would be "sqlpackage /a:publish /sf:db1.dacpac /tsn:localhost /tdn:db1" to publish dacpac "db1" to a database "localhost\db1". For other publish options type "sqlpackage /a:publish /?" to get the list of applicable actions on the command line.
Using Powershell you can deploy to either to Standard On-premise SQL Instances or to Azure SQL Instance. The connection string and other properties will be fed from the PublishProfile
$PublishPath = "Path for the log"
$dacpac = "dbname.dacpac"
$publishProfile = "dbname.xml" # Publish profile xml for Publishing the database project
# Generate Deploy Report
$DeployReport = ".\sqlpackage.exe /Action:DeployReport /Sourcefile:$dacpac `
/pr:'$publishProfile' /outputpath:$PublishPath"
Invoke-Expression $DeployReport
# Generate Script Report
$GenerateScript = ".\sqlpackage.exe /Action:Script /Sourcefile:$dacpac `
/pr:'$publishProfile' /outputpath:$PublishPath"
Invoke-Expression $GenerateScript
# Database Publish
$publish = ".\sqlpackage.exe /Action:Publish /Sourcefile:$dacpac `
/pr:'$publishProfile'"
Invoke-Expression $publish | Out-File $PublishPath
I have managed to create an instance and ssh into it. However, I have couple of questions regarding the Google Compute Engine.
I understand that I will be charged for the time my instance is running. That is till I exit out of the instance. Is my understanding correct?
I wish to run some batch job (java program) on my instance. How do I make my instance stop automatically after the job is complete (so that I don't get charged for the additional time it may run)
If I start the job and disconnect my PC, will the job continue to run on the instance?
Regards,
Asim
Correct, instances are charged for the time they are running. (to the minute, minimum 10 minutes). Instances run from the time they are started via the API until they are stopped via the API. It doesn't matter if any user is logged in via SSH or not. For most automated use cases users never log in - programs are installed and started via start up scripts.
You can view your running instances via the Cloud Console, to confirm if any are currently running.
If you want to stop your instance from inside the instance, the easiest way is to start the instance with the compute-rw Service Account Scope and use gcutil.
For example, to start your instance from the command line with the compute-rw scope:
$ gcutil --project=<project-id> addinstance <instance name> --service_account_scopes=compute-rw
(this is the default when manually creating an instance via the Cloud Console)
Later, after your batch job completes, you can remove the instance from inside the instance:
$ gcutil deleteinstance -f <instance name>
You can put halt command at the end of your batch script (assuming that you output your results on persistent disk).
After halt the instance will have a state of TERMINATED and you will not be charged.
See https://developers.google.com/compute/docs/pricing
scroll downn to "instance uptime"
You can auto shutdown instance after model training. Just run few extra lines of code after the model training is complete.
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('compute', 'v1', credentials=credentials)
# Project ID for this request.
project = 'xyz' # Project ID
# The name of the zone for this request.
zone = 'xyz' # Zone information
# Name of the instance resource to stop.
instance = 'xyz' # instance id
request = service.instances().stop(project=project, zone=zone, instance=instance)
response = request.execute()
add this to your model training script. When the training is complete GCP instance automatically shuts down.
More info on official website:
https://cloud.google.com/compute/docs/reference/rest/v1/instances/stop
If you want to stop the instance using the python script, you can follow this way:
from google.cloud.compute_v1.services.instances import InstancesClient
from google.oauth2 import service_account
instance_client = InstancesClient().from_service_account_file(<location-path>)
zone = <zone>
project = <project>
instance = <instance_id>
instance_client.stop(project=project, instance=instance, zone=zone)
In the above script, I have assumed you are using service-account for authentication. For documentation of libraries used you can go here:
https://googleapis.dev/python/compute/latest/compute_v1/instances.html
I am using SQL Server Integration Services (SSIS) in SQL Server Business Intelligent Development Studio.
I need to do a task that is as follows. I have to read from a source database and put it into a destination flat file. But at the same time the source database should be configurable.
That means in the OLEDB Connection Manager, the connection string should change dynamically. This connection string should be taken from a configuration/XML/flat file.
I read that I can use variables and expressions to change the connection string dynamically. But how do I read the connection string value from a config/XML/flat file and set the variable?
This part I am unable to do. Is this the right way to achieve this? Can we add web.config files to an SSIS project?
First add a variable to your SSIS package (Package Scope) - I used FileName, OleRootFilePath, OleProperties, OleProvider. The type for each variable is "string". Then I create a Configuration file (Select each variable - value) - populate the values in the configuration file - Eg: for OleProperties - Microsoft.ACE.OLEDB.12.0; for OleProperties - Excel 8.0;HDR=, OleRootFilePath - Your Excel file path, FileName - FileName
In the Connection manager - I then set the Properties-> Expressions-> Connection string expression dynamically eg:
"Provider=" + #[User::OleProvider] + "Data Source=" + #[User::OleRootFilePath]
+ #[User::FileName] + ";Extended Properties=\"" + #[User::OleProperties] + "NO \""+";"
This way once you set the variables values and change it in your configuration file - the connection string will change dynamically - this helps especially in moving from development to production environments.
Some options:
You can use the Execute Package Utility to change your datasource, before running the package.
You can run your package using DTEXEC, and change your connection by passing in a /CONNECTION parameter. Probably save it as a batch so next time you don't need to type the whole thing and just change the datasource as required.
You can use the SSIS XML package configuration file. Here is a walk through.
You can save your configrations in a database table.
Here's some background on the mechanism you should use, called Package Configurations: Understanding Integration Services Package Configurations.
The article describes 5 types of configurations:
XML configuration file
Environment variable
Registry entry
Parent package variable
SQL Server
Here's a walkthrough of setting up a configuration on a Connection Manager: SQL Server Integration Services SSIS Package Configuration - I do realize this is using an environment variable for the connection string (not a great idea), but the basics are identical to using an XML file. The only step(s) you have to change in that walkthrough are the configuration type, and then a path.
Goto Package properties->Configurations->Enable Package Configurations->Add->xml configuration file->Specify dtsconfig file->click next->In OLEDB Properties tick the connection string->connection string value will be displayed->click next and finish package is hence configured.
You can add Environment variable also in this process
These answers are right, but old and works for Depoloyement Package Model.
What I Actually needed is to change the server name, database name of a connection manager and i found this very helpful:
https://www.youtube.com/watch?v=_yLAwTHH_GA
Better for people using SQL Server 2012-2014-2016 ... with Deployment Project Model