We are a team working on a bunch of SSIS packages, which we share using version control (SVN). We have three ways of saving sensitive data in these packages :
not storing them at all
storing them with a user key
storing them with a password
However, each of these options is inconvenient while testing packages saved and committed by an other developer. For each such package, one has to update the credentials, no matter how the sensitive data was persisted.
Is there a better way to collaborate on SSIS packages?
Since my workplace uses file deployment, I use "Don't save sensitive" In order to make development easier, we also store config files with the packages in our version control system, and the connection strings for the development environment are stored in the config files. The config files are also stored in a commonly named folder, so if I retrieve the config files into my common config file area, then I can open any of our project packages and they will work for me for development. When the packages are deployed, they are secured by the deployment team on a machine where developers do not have access and the config file values for connection strings are changed to match the production environment.
We do somthing similar using database deployment. Each enviroment has a configuration database, and every package references a single xml config file in a common file path on every server/workstation, e.g., "c:\SSISConfig". This xml config file has one entry that points to the appropriate config database for that environment. All of the rest of the SSIS configs are stored in that config database. The config database in production is only accessible by the admin group, the developers do not have access. When new packages and configurations are deployed to prod, connection strings are updated by the admin group. The packages are all set to "Dont save sensitive".
Related
First let me give some background
We have our own VPS, so we do not wish to use Azure to host our web applications.
We have already successfully created a CI/CD pipeline to our VPS by installing an agent on it for a .NET Core project.
We use Azure DevOps (formerly known as VSTS) to host our code in GIT and handle our backlogs and CI/CD pipelines.
We have several .NET Framework projects where we use XTD transforms to transform our web.config files on delivery/deployment to have the correct connection strings and other configuration properties.
This makes it possible to pull the master branch from our remote repo and have it working in seconds on a previously unused (for this application) development environment without the need for any configuration.
Now to get to my question
The master branch of the .NET Core project for which we already have the CI/CD pipeline in place holds the configuration in the json files for the staging environment it is continuously delivered to. When a developer pull the master branch, he/she first needs to configure these to suite the local debug environment.
This is an undesirable situation for us.
How can we make it so that if we use .NET Core we can use a mechanism that will allow us to have the project work on a local debug environment without any configuration and in the CI/CD pipeline?
What have we already tried?
we have found that we can have multiple versions of the appsettings.json file for the different environments like appsettings.debug.json and than in the static method CreateWebhost of the Program class we can call on or the other. But how we can automate this is something that we haven't been able to figure out or find documentation about.
Okay, so here are some options you can take advantage of TODAY. (there are im sure more options/approaches)
Option A
Configure the master branch to have appsetting.development.json with connection string to DEV database( or lowest environment)
remove any connection string from appsettings.json
Merge master accordingly.
Create environment variables on each of the backend servers for the connection string; ex, system environment variable named ConnectionStrings:cartDB with connection string to the database for the environment for which the backend server used.
The result of this will be that when running using DEVELOPMENT as the environment variable, then it will be able to connect to database everyone can access.
However, since all OTHER web servers have environment variables with connection string, they will take highest level of precedence, and will therefore be the values set when calling something such as
string connectionString = Configuration.GetConnectionString("cartDB");
This will satisfy the requirements you mentioned above.
Option B:
Configure the master branch to have appsetting.development.json with connection string to DEV database( or lowest environment)
remove any connection string from appsettings.json
Place appsetting.staging.json, appsettings.prod.json in source control, and set environment name variable in web servers. :/ not the best of options/advised against.
(its worth mentioning since I have seen this happen, we all have)
Option C
Add appsetting.staging.json, appsettings.prod.json to source control and use a token in place of the connection string value. Then, leverage some type of Tokenization Task to replace those tokens with the appropriate values.
I was looking at a README file that raised some questions about database persistence on Openshift.
Note: Every time you push, everything in your remote repo dir gets recreated
please store long term items (like an sqlite database) in the OpenShift
data directory, which will persist between pushes of your repo.
The OpenShift data directory is accessible relative to the remote repo
directory (../data) or via an environment variable OPENSHIFT_DATA_DIR.
https://github.com/ryanj/nodejs-custom-version-openshift/blob/master/README#L24
However, I could find no confirmation of this on the Openshift website. Is this README out of date? I'd rather not test this, so it would be much appreciated if anyone had any firsthand knowledge they'd be willing to share.
Yep, that readme file is up to date regarding SQLite. All gears have SQLite installed on them. Data should be stored in the persistent storage directory on your gear. This does not apply to MySQL/MongoDB/PostgreSQL as those databases are add-on cartridges pre-configured to use persistent storage, whereas SQLite is simply installed and available for use.
See the first notice found in the OpenShift Origin documentation here: https://docs.openshift.org/origin-m4/oo_cartridge_guide.html
Specifically:
Cartridges and Persistent Storage: Every time you push, everything in
your remote repo directory is recreated. Store long term items (like
an sqlite database) in the OpenShift data directory, which will
persist between pushes of your repo. The OpenShift data directory can
be found via the environment variable $OPENSHIFT_DATA_DIR.
The official OpenShift Django QuickStart shows the design pattern you should follow for adding SQLite to your application via the deploy action hook. See: https://github.com/openshift/django-example/blob/master/.openshift/action_hooks/deploy
What is the 'best practices' way to configure connections in SSIS 2012 project (that will be deployed to the server)? Documentation and Google shows multiple ways to accomplish this, but which way takes full advantage of the 2012 project/deployment model and is easily customizable, maintainable etc?
Consider a project (NorthwindETL) with three packages with each package referencing localhost.Northwind. On SSIS server, there is a project 'NorthwindETL', and an Environment 'Dev'.
To configure the connection, I have the following options
IN BIDS: Hard code the connection (via connection manager) manually in each package. IN SSIS: Configure the SSISDB 'NorthwindETL' project, under the connection manager tab, modify the Northwind connection string, once for each package.
IN BIDS: Using the Connection Manager 'Parameterize…' option, create a project (or package) parameter to specify the connection string (Northwind_Conn). IN SSIS: Configure the SSISDB 'NorthwindETL' project, specify the Northwind_Conn parameter value.
IN BIDS: Create a project level Connection Manager (Project_Northwind_Conn). IN SSIS: Configure the Configure the SSISDB 'NorthwindETL' project, under the connection manager tab, modify the 'Project_Northwind_Conn' connection string.
IN SSIS: Create an Environment on SSISDB called 'DEV'. In the 'Dev' environment properties, under variables, create a variable 'Env_Northwind_Conn'. Configure the NorthwindETL project, set 'Northwind_Conn' to the environmental variable 'Env_Nothwind_Conn'
(Also, I would prefer a solution that allows us to specify items separately such as InitialCatalog and Server, but this is not necessary. Although the connection manager allows you to modify the InitialCatalog and Server properties, this does not seem to actually modify the ConnectionString.)
For connection managers that are to be used across all the packages, typically database connections, the 2012 release gives us Project Connection Manager (option 3). I find that to be the most enjoyable for connection managers as when I go to apply configuration, I'm applying it once at the project level instead of once per package.
Unless you run your dev/test/prod ETL from the same dedicated server, I am not a fan of naming my SSIS Environment Variables as such. The primary reason is that my deployment script then has to have the intelligence built into it to not only switch server names per tier, but also the environment name. Just makes more opportunity for the dumb to enter.
I'm also a fan of just creating an empty Folder within the SSISDB, call it Configs and then establish my SSIS Environment variables there. All projects then reference that folder's variables. At my clients, it's generally been the case that they're all referencing the same Sales database so it seems like more work for me to have to maintain N configurations to satisfy N projects instead of having a single shared configuration repository.
Script everything. Click the scroll icon as you create and assign your configurations. It's going to make it a far easier row to hoe when you need to migrate from one environment to the next.
I am just starting out with WEBMethods. I am starting on a project that will need to poll a drive on my companies M: drive location. The file will arrive randomly from the mainframe and I will need to have WEBMethods some how pulled the file from the drive location.
Once I have to move the file from one location to another before I start my parsing of the file.
If I had more code I would post it, but WEBMethods is new and so far I actually have not writen any code in WEBMethods but I am extremely code with Java.
Drive location:
M:\tempTest\NewDriveLocation\ThisIsTheFileINeed
I need to be able to have a transform that pulls in a file from any directly location on Friday. I have an input retieve on my MAP but have not figured out how to enter the file path so that it can find the file.
Software AG's webMethods Integration Server has a built-in feature called a File Polling Port, which you can configure to monitor a local or network shared directory for new files. The Integration Server Administrator's Guide instructions for how to set up a File Polling Port are as follows:
A file polling port periodically polls a monitoring directory for the arrival of files and
then performs special processing on them. When it detects a new file, the server copies
the file to a working directory, then runs a special file processing service against the file.
The service might parse, convert, and validate the file then write it to the file system. This
service, which you write, is the only service that can be invoked through this port. You
can limit the files the server accepts by filtering for specific file names.
For file polling to work, you must do the following:
Set up the Monitoring Directory on Integration Server. Other directories used for file
polling are automatically created by Integration Server.
Write a file processing service and make it available to Integration Server. See
webMethods Service Development Help and the Flat File Schema Developer's Guide for
examples of such services.
Set up the file polling port on Integration Server.
Use the following procedure to add a file polling port to Integration Server.
Open Integration Server Administrator if it is not already open.
In the Security menu of the Navigation panel, click Ports.
Click Add Port.
In the Add Port area of the screen, select webMethods/FilePolling.
Click Submit. Integration Server Administrator displays a screen requesting
information about the port.
Under Package, enter the following information:
Package Name - The package associated with this port.
When you enable the package, the server enables the port.
When you disable the package, the server disables the port.
If you are performing special file handling, specify the
package that contains the services that perform that
processing. If you want to process flat files from this port,
select WmFlatFile,which contains built-in services you can
use to process flat files.
Note: If you replicate this package, whether to a server on
the same machine or a server on a separate machine, a file
polling port with the same settings is created on the target
server. If a file polling port already exists on the target
server, its settings remain intact. If the original and target
servers reside on the same machine, they will share the
same monitoring directory. If the target server resides on
another machine, by default, another monitoring directory
will be created on the target server's machine.
Alias - An alias for the port. An alias must be between 1 and 255
characters in length and include one or more of the
following: ASCII characters, numbers, underscore (_),
period (.), and hyphen (-).
Description - A description of the port.
Under Polling Information, enter the following information:
Monitoring Directory - Directory on Integration Server that you want to
monitor for files.
Working Directory (optional) - Directory on Integration Server to which the server
should move files for processing after they have been
identified in the Monitoring Directory. Files must meet
age and file name requirements before being moved to
the Working Directory. The default sub-directory,
MonitoringDirectory..\Work, is automatically created
if no directory is specified.\
Completion Directory (optional) - Directory on Integration Server to which you want files
moved when processing is completed in the Monitoring
Directory or Working Directory. The default sub-directory,
MonitoringDirectory..\Done, is automatically created
if no directory is specified.
Error Directory (optional) - Directory on Integration Server to which you want files
moved when processing fails. The default subdirectory,
MonitoringDirectory..\Error, is
automatically created if no directory is specified.
File Name Filter (optional) - The file name filter for files in the Monitoring Directory.
The server only processes files that meet the filter
requirements. If you do not specify this field, all files
will be polled. You can specify pattern matching in this
field.
File Age (optional) - The minimum age (in seconds) at which a file in the
Monitoring Directory can be processed. The server
determines file age based on when the file was last
modified on the monitoring directory. You can adjust
this age as needed to make sure the server does not
process a file before the entire file has been copied to
the Monitoring Directory. The default is 0.
Content Type - Content type to use for the file. The server uses the
content handler associated with the content type
specified in this field. If no value is specified, the server
performs MIME mapping based on the file extension.
Allow Recursive Polling - Whether Integration Server is to poll all sub-directories
in the Monitoring Directory. Select Yes or No.
Enable Clustering Whether Integration Server should allow clustering in
the Monitoring Directory. Select Yes or No.
Number of files to process per interval (optional) -
Specifies the maximum number of files that the file
polling listener can process per interval. When you
specify a positive integer, the file polling listener
processes only that number of files from the
monitoring directory. Any files that remain in the
monitoring directory will be processed during
subsequent intervals. If no value is specified, the
listener processes all of the files in the monitoring
directory.
Under Security, in the Run services as user parameter, specify the user name you want
to use to run the services assigned to the file polling directory. Click to lookup and
select a user. The user can be an internal or external user.
Under Message Processing, supply the following information:
Enable - Whether to enable (Yes) or disable (No) this file polling
port.
Processing Service - Name of the service you want Integration Server to
execute for polled files. The server executes this service
when the file has been copied to the Working directory.
This service should be the only service available from
this port.
Important! If you change the processing service for a file
polling port, you must also change the list of services
available from this port to contain just the new service.
See below for more information.
File Polling Interval - How often (in seconds) you want Integration Server to
poll the Monitoring Directory for files.
Log Only When Directory Availability Changes -
If you select No (the default), the listener will log a
message every time the monitoring directory is
unavailable.
If you select Yes, the listener will log a message in
either of the following cases:
The directory was available during the last polling
attempt but not available during the current
attempt
The directory was not available during the last
polling attempt but is available during the current
attempt
Directories are an NFS Mounted File System - For use on a UNIX system where the monitoring
directory, working directory, completion directory,
and/or error directory are network drives mounted on
the local file system.
If you select No (the default), the listener will call the
Java File.renameTo() method to move the files from the
monitoring directory to the working directory, and
from the working directory to the completion and/or
error directory.
If you select Yes, the listener will first call the Java
File.renameTo() method to move the files from the
monitoring directory. If this method fails, the listener
will then copy the files from the monitoring directory
to the working directory and delete the files from the
monitoring directory. This operation will fail if either
the copy action or the delete action fails. The same
behavior applies when moving files from the working
directory to the completion and/or error directory.
Cleanup Service (Optional) - The name of the service that you want to use to clean
up the directories specified under Polling Information.
Cleanup At Startup - Whether to clean up files that are located in the
Completion Directory and Error Directory when the file
polling port is started.
Cleanup File Age (Optional) - The number of days to wait before deleting processed
files from your directories. The default is 7 days.
Cleanup Interval (Optional) - How often (in hours) you want Integration Server to
check the processed files for cleanup. The default is 24
hours
Maximum Number of Invocation Threads -
The number of threads you want Integration Server to
use for this port. Type a number from 1-10. The default
is 10.
Click Save Changes.
Make sure the port's access mode is properly set and that the file processing service is
the only service accessible from the port.
In the Ports screen, click Edit in the Access Mode field for the port you just created.
Click Set Access Mode to Deny by Default.
Click Add Folders and Services to Allow List.
Type the name of the processing service for this port in the text box under Enter
one folder or service per line.
Remove any other services from the allow list.
Click Save Additions.
Note: If you change the processing service for a file polling port, remember to
change the Allow List for the port as well. Follow the procedure described above
to alter the allowed service list.
If you change the processing service for a file polling port, remember to change
the Allow List for the port as well. Follow the procedure described above to alter
the allowed service list.
The Processing Service referenced above is a service which you must develop.
If you are processing XML files with the File Polling Port, the file will be parsed prior to invoking your service, so you should create a service which has a single input argument of type object called node (which is the parsed XML document). You can then use the pub.xml services in the WmPublic package (such as pub.xml:xmlNodeToDocument to convert the node to an IData document) to process the provided node object. Refer to the Integration Server Built-In Services Reference for details on the pub.xml services.
If you are processing flat files (which is anything other than XML in webMethods), the File Polling Port will invoke your service with a java.io.InputStream object from which you can read the file contents, so you should create a service which has a single input argument of type object called ffdata. You can then use the pub.io services in the WmPublic package (such as pub.io:streamToBytes to read all data in the stream to a byte array) or the pub.flatFile services in the WmFlatFile package (such as pub.flatFile:convertToValues to convert ffdata to an IData document) to process the provided ffdata object. Refer to the Integration Server Built-In Services Reference for details on the pub.io services, and the Flat File Schema Developer's Guide for details on the pub.flatFile services.
If both XML and flat files are being written to the monitored directory, you can either write a service which optionally accepts both a node and ffdata object and check which one exists in the pipeline at runtime and process accordingly, or you can create two File Polling Ports which monitor the same directory but check for different file extensions (ie. *.xml and *.txt respectively) using the File Name Filter setting on the port.
If you want to poll a Windows file share, you can specify the directory using a UNC file path (such as \\server\directory) on the File Polling Port.
Also, you need to make sure the user account under which Integration Server executes has appropriate file access rights to the various directories configured on the File Polling Port.
I have a windows service that reads from app.config
I want some settings to be encrypted, however, I don't want to use the ProtectedConfigurationProvider classes provided in .NET because they encrypt files based on the machine they are running on using DPAPI.
What I wanted was a way for our administrator to deploy the config file already encrypted to many machines and have each machine decrypt them when needed.
I don't want to hardcode a password into the assembly either so I'm not sure how I can go about this.
Perhaps your central server can maintains a database of the private keys for all the end points, then it could use the specific machine key for that.
A shared private key for all the machines is not going to be that secure.
You can use user-level RSA keys and export them to each machine. then the config file can be encrypted ahead of time, and the target machines already have the key.