Using Newtonsoft.Json in a TFS release definition from powershell - json

Folks,
We are using TFS 2017 on-premise and are working toward provisioning/deploying to an Azure website and database. Our Azure database provisioning/updating task works well, but until we can automate the website portion part, we have an interim step where we copy the web package from the build into a staging share.
This step runs on a Deployment Group under the TFS vstsagent as follows:
Create the temp folder
Expand the zip file into the temp folder
Retrieve the Out folder under the expanded zip file
Move out folder to root of temp folder
Merge the appsettings.json file with the master.appsettings.json file for the environment.
Re-zip the web package
Copy the web package to staging destination share under PRJ\build#
Step 5) is performed using the merge feature of Newtonsoft.Json.
# Load the Newtonsoft.Json dll
$asmNewtonsoftJson = [Reflection.Assembly]::LoadFile([io.path]::combine($FolderToZip, "Newtonsoft.Json.dll"))
# Read the appsettings.json file and store as a JObject
$appSettingsPath = [io.path]::combine($folderToZip, "appsettings.json")
$appSettingsJson = (Get-Content $appSettingsPath | Out-String)
$appSettings = [Newtonsoft.Json.Linq.JObject]::Parse($appSettingsJson)
# Read the master appsettings.json file from the env and store as a JObject
$appSettingsMasterPath = [io.path]::combine($AppSettingsMasterFolderPath, $Environment + ".appsettings.json")
$appSettingsMasterJson = (Get-Content $appSettingsMasterPath | Out-String)
$appSettingsMaster = [Newtonsoft.Json.Linq.JObject]::Parse($appSettingsMasterJson)
# Merge the master appsettings.json file into the appsettings.json file
$jms = New-Object Newtonsoft.Json.Linq.JsonMergeSettings
$jms.MergeArrayHandling = [Newtonsoft.Json.Linq.MergeArrayHandling]::Merge
$appSettings.Merge($appSettingsMaster, $jms)
# Write out the updated appsettings.json file
[io.File]::WriteAllText($appSettingsPath, $appSettings.ToString())
Note: $folderToZip contains the expanded contents of the web package. This also contains the Newtonsoft.Json dll (and presumably its dependencies).
The above code works fine in the Windows Powershell ISE app, but fails when run in the TFS release definition task.
It fails at step 5) with the following error:
2018-10-11T19:29:39.1790546Z ##[error]Unable to find type [Newtonsoft.Json.Linq.JObject].
At C:\vstsagent\A1\_work\r7\a\<path>\drop\Build\mymergescript.ps1:84 char:20
$appSettings = [Newtonsoft.Json.Linq.JObject]::Parse($appSettings)
CategoryInfo : InvalidOperation: (Newtonsoft.Json.Linq.JObject:TypeName) [], RuntimeException
FullyQualifiedErrorId : TypeNotFound
It appears that the Newtonsoft.Json.dll isn't getting loaded properly when deployed by the TFS vstsagent.
Any help here is appreciated.

Related

Failed loading positionFile: while using TAILDIR Source in flume i am getting error

I working on Flume to append the data from a local directory to HDFS using Flume Source TAILDIR.
My use case is to do Delta Load If the new line comes in the source file in local dir so that will append in hdfs.
This is my Flume Conf file :
#configure the agent
agent.sources=r1
agent.channels=k1
agent.sinks=c1
agent.sources.r1.type=TAILDIR
agent.sources.r1.positionFile = /home/flume/Documents/taildir_position.json
agent.sources.r1.filegroups=f1
agent.sources.r1.filegroups.f1=/home/flume/Documents/spooldir/
agent.sources.r1.batchSize = 20
agent.sources.r1.writePosInterval=2000
agent.sources.r1.maxBackoffSleep=5000
agent.sources.r1.fileHeader = true
agent.sources.r1.channels=k1
agent.channels.k1.type=memory
agent.channels.k1.capacity=10000
agent.channels.k1.transactionCapacity=1000
agent.sinks.c1.type=hdfs
agent.sinks.c1.channel=k1
agent.sinks.c1.hdfs.path=hdfs://localhost:8020/flume_sink
agent.sinks.c1.hdfs.batchSize = 1000
agent.sinks.c1.hdfs.rollSize = 268435456
agent.sinks.c1.hdfs.writeFormat=Text
while running flume command : flume-ng agent -n agent -c conf -f /home/swechchha/Documents/flumereal.conf
I am getting error
I am getting error to load JSON file.
Here is the code. It crashes at the line 110. Please make sure that flume user has access to that JSON file and that the file is correctly formatted.
The Flume.conf mentioned in Question Statement is having a problem.
TAILDIR SOURCE: Watch the specified files, and tail them in nearly real-time once detected new lines appended to each files. If the new lines are being written, this source will retry reading them in wait for the completion of the write.
While writing filegroups property directory may contain multiple files in this case it should be mentioned like directory path/ .filestype.
agent.sources.r1.filegroups.f1=/home/flume/Documents/spooldir/.*txt.*
Then run flume.conf and check the result it will work fine.

jmeter.functions.FileToString not finding file location

In JMeter I am passing multiple JSON inputs as body, Variable name is defined as JSON_FILE and coming from CSV Data Config
${__FileToString(${__eval(${JSON_FILE})}.json,,)}
CSV Data
designO1015643320
.
.
designO1077673985
designO1088516727
Running load test from Jmeter UI works fine, but running as mvn project is giving error about FileNotFoundException even though .csv file and .json files are in same folder as .jmx file
Error from .jmx.log:
WARN - jmeter.functions.FileToString: Could not read file: designO1015643320.json File 'designO1015643320.json' does not exist java.io.FileNotFoundException: File 'designO1015643320.json' does not exist
Response in .jtl:
httpSample t="4" lt="0" ts="1508530091457" s="false" lb="CreateDesign_PUT" rc="Non HTTP response code: org.apache.jorphan.util.JMeterStopThreadException" rm="Non HTTP response message: End of sequence" tn="Design_APIs 1-1" dt="text" by="1822" ng="1" na="1"/>
JMeter GUI default relative path is the bin folder
Relative paths are resolved relative to the current working directory (which defaults to the bin/ directory).
Maven search in different default path for files src/test/jmeter directory
See guide:
in the src/test/jmeter directory. When running the project, the JMeter Maven plugin searches for tests to run in this directory.
And you can find this path dynamically
I heard Groovy is a new black so I would recommend replacing your __FileToString() function with __groovy() function, the Groovy equivalent of dynamically getting the file path relative to Maven's plugin current working directory would be something like:
${__groovy(new File(org.apache.jmeter.services.FileServer.getFileServer().getBaseDir() + System.getProperty('file.separator') + vars.get('JSON_FILE') + '.json').text,)}
See JavaDoc on FileServer class for more details.

cx_freeze to access .json files

I have created an application for windows using pythons cx_freeze module. The application runs the openpyxl module which runs fine for the script but when frozen it fails to find the .constants.json files. The following error is displayed.
FileNotFoundError: [Errno 2] No such file or directory: 'C:....\exe.win-amd64-3.4\library.zip\openpyxl.constants.json'
I have found a fix for this (https://cx-freeze.readthedocs.org/en/latest/faq.html#using-data-files) detailed below :
def find_data_file(filename):
if getattr(sys, 'frozen', False):
# The application is frozen
datadir = os.path.dirname(sys.executable)
else:
# The application is not frozen
# Change this bit to match where you store your data files:
datadir = os.path.dirname(__file__)
return os.path.join(datadir, filename)
The question I have is where do I paste this code? Does it go in the setup.py file? Or somewhere else?

convert VMX to OVF using OVFtool

I am trying to convert VMX to OVF format using OVFTool as below, however it gives error:
C:\Program Files\VMware\VMware OVF Tool>ovftool.exe
vi://vcenter.com:port/folder/myfolder/abc.vmx abc.ovf
Error: Failed to open file: https://vcenter.com:port/folder/myfolder/abc.vmx
Completed with errors
Please let me know if you have any solution.
I had a similar situation in vmware fusion trying to use a .vmx that was probably created on windows. I could boot the VM, but any attempt to export the machine with ovftool or use vmware-vdiskmanager bombed out with:
Error: Failed to open disk: source.vmdk
Completed with errors
the diskname was totally valid, path was valid, permissions were valid, and the only clue was running ovftool with:
ovftool --X:logToConsole --X:logLevel=verbose source.vmx dest.ova
Opening VMX source: source.vmx
verbose -[10C2513C0] Opening source
verbose -[10C2513C0] Failed to open disk: ./source.vmdk
verbose -[10C2513C0] Exception: Failed to open disk: source.vmdk. Reason: Disk encoding error
Error: Failed to open disk: source.vmdk
as others suggested, i took a peek in the .vmdk. therein i found 3 other clues:
encoding="windows-1252"
createType="monolithicSparse"
# Extent description
RW 16777216 SPARSE "source.vmdk"
so first i converted the monolithicSparse vmdk to "preallocated virtual disk split in 2GB files":
vmware-vdiskmanager -r source.vmdk -t3 foo.vmdk
then i could edit the "foo.vmdk" to change the encoding, which now looks like:
encoding="utf-8"
createType="twoGbMaxExtentFlat"
# Extent description
RW 8323072 FLAT "foo-f001.vmdk" 0
RW 8323072 FLAT "foo-f002.vmdk" 0
RW 131072 FLAT "foo-f003.vmdk" 0
and finally, after fixing up the source.vmx:
scsi0:0.fileName = "foo.vmdk"
profit:
ovftool source.vmx dest.ova
...
Opening VMX source: source.vmx
Opening OVA target: dest.ova
Writing OVA package: dest.ova
Transfer Completed
Completed successfully
I had a similar problem with OVFTool trying to export to OVF format.
Export failed: Failed to open file: C:\Virtual\test\test.vmx.
First, I opened .VMX file in editor (it's a text file) and made sure that settings like
scsi0:0.fileName = "test.vmdk"
nvram = "test.nvram"
extendedConfigFile = "test.vmxf"
mention proper file names.
Then I noticed this line:
.encoding = "windows-1251"
This is Cyrillic code page, so I modified it to use Western code page
.encoding = "windows-1252"
Then, running OVFTool gave a different error
Export failed: Failed to open disk: test.vmdk.
To fix it I had to open .VMDK file in HEX editor (because it's usually a big binary file), found there the string
encoding = "windows-1251"
(it's somewhere in the beginning of the file), and replaced "1251" with "1252".
And it did the trick!
In my case, was needed repair the disk 'abc.vmdk' before convert the 'abc.vmx' to 'abc.ovf'.
Use this for Linux:
$ /usr/bin/vmware-vdiskmanager -R /home/user/VMware/abc.vmdk
Look this link https://kb.vmware.com/s/article/2019259 for resolved issue in Windows and Linux
Try to run as described below.
C:\Program Files\VMware\VMware OVF Tool>ovftool C:\Win-Test\Win-Test.vmx(location of your vmx file) C:\Win-Test\win-test.ovf (destination)
Maybe ovftool is unable to recognize the path you are giving.
Try with following command:
ovftool --eula#=[path to eula] --X:logToConsole --targetType=OVA --compress=9 vi://[username]:[ESX address] [target address]
Once you provide the ESX address, it will list down the folders you have created in your ESX box. Then you can trigger the command above mentioned again with appending folder name.
If no folder hierarchy present in your box, then it will simply list down vm names.
Retry the same command appending [foldername]/[vmname no vmx file name required]
ovftool --eula#=[path to eula] --X:logToConsole --targetType=OVA --compress=9 vi://[username]:[ESX address]/[foldername if exist]/[vmname no vmx file name required] [target address]
I had this same exact issue. In my case I opened up the VMX file and dropped the IDE and sound controllers from the file and saved. I was then able to convert everything to an OVA using the tool with the standard syntax.
e.g. I dropped:
ide1:0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
and:
sound.present = "TRUE"
sound.fileName = "-1"
sound.autodetect = "TRUE"
This allowed me to convert the file like normal.
For me opening the .vmx and deleting the following line worked:
sata0:1.deviceType = "cdrom-image"
In my case, this works:
ide1:0.present = "TRUE"
ide1:0.deviceType = "cdrom-image"
I did change true to false and works fine, as cdrom-image not exist, this change permit the format conversion.
if your goal is to move a windows based vm to virtual box you only need to:
uninstall vmware tools from the guest vm
shut down the machine
copy the hd to a new folder
create a new empty vm in virtualbox
mount the hd (the .vmdk file) in that vm
Easy and rapid to do.

How to publish DACPAC file to a SQL Server database project via SQLPackage.exe of SSDT?

I'm using SSDT for Visual Studio 2012 here and using its command-line tool SQLPackage.exe to publish a .dacpac file.
I want to publish that to an SQL Server database project. I'm trying to use parameter at this guide but cannot find ways how to do it.
How can I do that?
Are you trying to publish to a Database, or create a database project from a dacpac? These are two different things.
To create a database project based on a dacpac, create a new SQL Server Database Project in Visual Studio. Then right-click on the project in Solution Explorer and choose "Import -> Data-tier Application (*.dacpac)" and select your dacpac. That will convert the contents of the dacpac into scripts in the project, and if you choose "Import database settings" the database options will be set based on the settings in the dacpac. Unfortunately there's no way to do this via a command line tool that I'm aware of.
To use SqlPackage.exe to publish to a database, the guide you linked to has all the options. A simple command would be "sqlpackage /a:publish /sf:db1.dacpac /tsn:localhost /tdn:db1" to publish dacpac "db1" to a database "localhost\db1". For other publish options type "sqlpackage /a:publish /?" to get the list of applicable actions on the command line.
Using Powershell you can deploy to either to Standard On-premise SQL Instances or to Azure SQL Instance. The connection string and other properties will be fed from the PublishProfile
$PublishPath = "Path for the log"
$dacpac = "dbname.dacpac"
$publishProfile = "dbname.xml" # Publish profile xml for Publishing the database project
# Generate Deploy Report
$DeployReport = ".\sqlpackage.exe /Action:DeployReport /Sourcefile:$dacpac `
/pr:'$publishProfile' /outputpath:$PublishPath"
Invoke-Expression $DeployReport
# Generate Script Report
$GenerateScript = ".\sqlpackage.exe /Action:Script /Sourcefile:$dacpac `
/pr:'$publishProfile' /outputpath:$PublishPath"
Invoke-Expression $GenerateScript
# Database Publish
$publish = ".\sqlpackage.exe /Action:Publish /Sourcefile:$dacpac `
/pr:'$publishProfile'"
Invoke-Expression $publish | Out-File $PublishPath