Converting Packer 1.6 vsphere-iso configuration code from JSON to HCL2 - json

With the release of Packer 1.6 came several depreciated fields in the vsphere-iso builder. From the looks of it, seems to be a format/type change because the fields actually still exists but just as properties it seems. An example of the changes are the following:
Working in Packer 1.5.6:
JSON
"disk_size": 123456,
"disk_thin_provisioned": true
"network": "VM Network",
"network_card": "vmxnet3"
Working in Packer 1.6.0:
JSON
"storage": [
{
"disk_size": 123456,
"disk_thin_provisioned": true
}
],
"network_adapters": [
{
"network": "VM Network",
"network_card": "vmxnet3"
}
]
The issue I have at the moment is I'm using Packer 1.6.0 and am trying to convert the above working JSON code to HCL2. I can't figure out the HCL2 syntax that supports the changes that were made in Packer 1.6.0.
I've tried the following:
network_adapters = {
network_card = "vmxnet3"
network = "VM Network"
}
Output:
An argument named "network_adapter" is not expected here.
network_adapters = (
network_card = "vmxnet3"
network = "VM Network"
)
Output:
Error: Unbalanced parentheses
on .\Packer\ConfigFileName.pkr.hcl line 19, in source "vsphere-iso"
"Test": 18: storage = ( 19: disk_thin_provisioned = true
Expected a closing parenthesis to terminate the expression.
network_adapters = [
network_card = "vmxnet3",
network = "VM Network"
]
Output:
Error: Missing item separator
on .\Packer\ConfigFileName.pkr.hcl line 19, in source "vsphere-iso"
"Test": 18: storage = [ 19: disk_thin_provisioned =
true,
Expected a comma to mark the beginning of the next item.
I've also tried several other permutations of different collection syntax together with no luck so far. Any suggestions or tips would greatly be appreciated

The correct syntax is the following:
network_adapters {
network_card = "vmxnet3",
network = "VM Network"
}
Note that it's not using an assignment operator = between network_adapters and {
Credit goes to SwampDragons over on the Packer forums for pointing this out.
If you're interested in knowing why: There was a change to how maps are treated in HCL2 back in May 2020 with the release of Packer 1.5.6
core/hcl2: Maps are now treated as settable arguments as opposed to blocks. For example tags = {} instead of tags {} [GH-9035]
Reference: https://github.com/hashicorp/packer/blob/master/CHANGELOG.md#156-may-1-2020

Related

Having problems using google cloud functions

message: "Build failed: *** Error compiling './main.py'...
File "./main.py", line 85
"prompt": prompt,
^
SyntaxError: invalid syntax; Error ID: 49c34848"
if user_request == "longer responses":
model_config = {
"engine": "gpt3",
"model": ["text-davinci-002", "text-davinci-003", "text-curie-001", "text-babbage-001", "text-ada-001",]
"prompt": prompt,
"temperature": 0.9, # set the temperature to a high value
"max_tokens": 300, # set the max_tokens filter to 160 tokens
It doesnt work. I tried add ing semicolons. It doesnt work
I think your code is incorrect, it remains } character to write correctly the Python Dict. I also removed a useless coma in the array :
if user_request == "longer responses":
model_config = {
"engine": "gpt3",
"model": ["text-davinci-002", "text-davinci-003", "text-curie-001", "text-babbage-001", "text-ada-001"]
"prompt": prompt,
"temperature": 0.9, # set the temperature to a high value
"max_tokens": 300 # set the max_tokens filter to 160 tokens
}

How to get object name when it contains DOTs, CURLY BRACES and HASHTAGs on JSONPath?

I have the following JSON structure, generated by Zabbix Discovery key, with the following data:
[{
"{#SERVICE.NAME}": ".WindowsService1",
"{#SERVICE.DISPLAYNAME}": ".WindowsService1 - Testing",
"{#SERVICE.DESCRIPTION}": "Application Test 1 - Master",
"{#SERVICE.STATE}": 0,
"{#SERVICE.STATENAME}": "running",
"{#SERVICE.PATH}": "E:\\App\\Test\\bin\\testingApp.exe",
"{#SERVICE.USER}": "LocalSystem",
"{#SERVICE.STARTUPTRIGGER}": 0,
"{#SERVICE.STARTUP}": 1,
"{#SERVICE.STARTUPNAME}": "automatic delayed"
},
{
"{#SERVICE.NAME}": ".WindowsService2",
"{#SERVICE.DISPLAYNAME}": ".WindowsService2 - Testing",
"{#SERVICE.DESCRIPTION}": "Application Test 2 - Slave",
"{#SERVICE.STATE}": 0,
"{#SERVICE.STATENAME}": "running",
"{#SERVICE.PATH}": "E:\\App\\Test\\bin\\testingApp.exe",
"{#SERVICE.USER}": "LocalSystem",
"{#SERVICE.STARTUPTRIGGER}": 0,
"{#SERVICE.STARTUP}": 1,
"{#SERVICE.STARTUPNAME}": "automatic delayed"
}]
So, what i want to do is: Use JSONPath to get ONLY the object that {#SERVICE.NAME} == WindowsService1...
The problem is, i am trying to create the JSONPath but it's giving me a couple of errors.
Here's what i tried, and what i discovered so far:
JSONPath:
$.[?(#.{#SERVICE.NAME} == '.WindowsService1')]
Error output:
jsonPath: Unexpected token '{': _$_v.{#SERVICE.NAME} ==
'.WindowsService1'
I also tried doing the following JSONPath, to match Regular Expression:
$.[?(#.{#SERVICE.NAME} =~ '^(.WindowsService1$)')]
It gave me the same error - So the problem is not after the == or =~ ...
What i discovered is, if i REMOVE the curly braces {}, the hashtag # and replace the dot . in "Service name" with _ (Underline), in JSONPath and in JSON data, it works, like this:
Data without # {} . :
[{
"SERVICE_NAME": ".WindowsService1",
[...]
JSONPath following new data structure:
$.[?(#.SERVICE_NAME == '.WindowsService1')]
But the real problem is, i need to maintain the original strucutre, with the curly braces, dots, and hashtags...
How can i escape those and stop seeing this error?
Thank you...
$.[?(#['{#SERVICE.NAME}'] == '.WindowsService1')]

Glue_version and python_version not working in terraform

Hellow everyone,
I am using terraform to create the glue job. Now AWS Glue now supports the ability to run ETL jobs on Apache Spark 2.4.3 (with Python 3).
I want to use this feature. but whenever i am making changes it is throwing error.
I am using
aws-cli/1.16.184.
Terraform v0.12.6
aws provider 2.29
resource "aws_glue_job" "aws_glue_job_foo" {
glue_version = "1"
name = "job-name"
description = "job-desc"
role_arn = data.aws_iam_role.aws_glue_iam_role.arn
max_capacity = 1
max_retries = 1
connections = [aws_glue_connection.connection.name]
timeout = 5
command {
name = "pythonshell"
script_location = "s3://bucket/script.py"
python_version = "3"
}
default_arguments = {
"--job-language" = "python"
"--ENV" = "env"
"--ROLE_ARN" = data.aws_iam_role.aws_glue_iam_role.arn
}
execution_property {
max_concurrent_runs = 1
}
}
But it is throwing error to me,
Error: Unsupported argument
An argument named "glue_version" is not expected here.
This Terraform issue has been resolved.
Terraform aws_glue_job now accepts a glue_version argument.
Previous Answer
With or without python_version in the Terraform command block, I must go to the AWS console to edit the job and set "Glue version". My job fails without this manual step.
Workaround #1
This issue has been reported and debated and includes a workaround.
resource "aws_glue_job" "etl" {
name = "${var.job_name}"
role_arn = "${var.iam_role_arn}"
command {
script_location = "s3://${var.bucket_name}/${aws_s3_bucket_object.script.key}"
}
default_arguments = {
"--enable-metrics" = ""
"--job-language" = "python"
"--TempDir" = "s3://${var.bucket_name}/TEMP"
}
# Manually set python 3 and glue 1.0
provisioner "local-exec" {
command = "aws glue update-job --job-name ${var.job_name} --job-update 'Command={ScriptLocation=s3://${var.bucket_name}/${aws_s3_bucket_object.script.key},PythonVersion=3,Name=glueetl},GlueVersion=1.0,Role=${var.iam_role_arn},DefaultArguments={--enable-metrics=\"\",--job-language=python,--TempDir=\"s3://${var.bucket_name}/TEMP\"}'"
}
}
Workaround #2
Here is a different workaround.
resource "aws_cloudformation_stack" "network" {
name = "${local.name}-glue-job"
template_body = <<STACK
{
"Resources" : {
"MyJob": {
"Type": "AWS::Glue::Job",
"Properties": {
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://${local.bucket_name}/jobs/${var.job}"
},
"ExecutionProperty": {
"MaxConcurrentRuns": 2
},
"MaxRetries": 0,
"Name": "${local.name}",
"Role": "${var.role}"
}
}
}
}
STACK
}
This has been released in version 2.34.0 of the Terraform AWS provider.
It looks like terraform uses python_version instead of glue_version
By using python_version = "3", you should be using glue version 1.0. Glue version 0.9 doesn't support python 3.

Scrapy multiple regular expressions in LinkExtractor seem to be not working

I've got my regular expressions inside a JSON file. This file gets loaded as a configuration for my spider. The spider creates one LinkExtractor with allow and deny regular expression rules.
I'd like to:
crawl and scrape product pages (scraping / parsing is NOT working)
crawl category pages
avoid general pages (about us, privacy, etc.)
It all works well on some shops, but not on others and I believe it's a problem of my Regular Expressions.
"rules": [
{
"deny": ["\\/(customer\\+service|ways\\+to\\+save|sponsorship|order|cart|company|specials|checkout|integration|blog|brand|account|sitemap|prefn1=)\\/"],
"follow": false
},
{
"allow": ["com\\/store\\/details\\/"],
"follow": true,
"use_content": true
},
{
"allow": ["com\\/store\\/browse\\/"],
"follow": true
}
],
URL patterns:
Products:
https://www.example.com/store/details/Nike+SB-Portmore-II-Solar-Canvas-Mens
https://www.example.com/store/details/Coleman+Renegade-Mens-Hiking
https://www.example.com/store/details/Mueller+ATF3-Ankle-Brace
https://www.example.com/store/details/Planet%20Fitness+18
https://www.example.com/store/details/Lifeline+Pro-Grip-Ring
https://www.example.com/store/details/Nike+Phantom-Vision
Categories:
https://www.example.com/store/browse/footwear/
https://www.example.com/store/browse/apparel/
https://www.example.com/store/browse/fitness/
Deny:
https://www.example.com/store/customer+service/Online+Customer+Service
https://www.example.com/store/checkout/
https://www.example.com/store/ways+to+save/
https://www.example.com/store/specials
https://www.example.com/store/company/Privacy+Policy
https://www.example.com/store/company/Terms+of+Service
Loading the rules from JSON inside my spider __init__
for rule in self.MY_SETTINGS["rules"]:
allow_r = ()
if "allow" in rule.keys():
allow_r = [a for a in rule["allow"]]
deny_r = ()
if "deny" in rule.keys():
deny_r = [d for d in rule["deny"]]
restrict_xpaths_r = ()
if "restrict_xpaths" in rule.keys():
restrict_xpaths_r = [rx for rx in rule["restrict_xpaths"]]
Sportygenspider.rules.append(Rule(
LinkExtractor(
allow=allow_r,
deny=deny_r,
restrict_xpaths=restrict_xpaths_r,
),
follow=rule["follow"],
callback='parse_item' if ("use_content" in rule.keys()) else None
))
If I do a pprint(vars(onerule.link_extractor)) I can see the Python regex correctly:
'deny_res': [re.compile('\\/(customer\\+service|sponsorship|order|cart|company|specials|checkout|integration|blog|account|sitemap|prefn1=)\\/')]
{'allow_domains': set(),
'allow_res': [re.compile('com\\/store\\/details\\/')],
{'allow_domains': set(),
'allow_res': [re.compile('com\\/store\\/browse\\/')],
Testing the regex in https://regex101.com/ seems to be fine as well (despite: I'm using \\/ in my JSON file and \/ in regex101.com)
In my spider logfile, I can see that the produce pages are being crawled, but not parsed:
2019-02-01 08:25:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.example.com/store/details/FILA+Hometown-Mens-Lifestyle-Shoes/5345120230028/_/A-6323521;> (referer: https://www.example.com/store/browse/footwear)
2019-02-01 08:25:47 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.example.com/store/details/FILA+D-Formation-Mens-Lifestyle-Shoes/5345120230027/_/A-6323323> (ref
Why does the spider not parse the product pages?
(same code, different JSON works on different shops)
After hours of debugging and testing, I figured that I had to change the order of the rules.
Products to scrape rule
Deny about us etc.
Categories to follow
Now it is working.
"rules": [
{
"allow": ["com\\/store\\/details\\/"],
"follow": true,
"use_content": true
},
{
"deny": ["\\/(customer\\+service|ways\\+to\\+save|sponsorship|order|cart|company|specials|checkout|integration|blog|brand|account|sitemap|prefn1=)\\/"],
"follow": false
},
{
"allow": ["com\\/store\\/browse\\/"],
"follow": true
}
],

How to keep appium capabilities in json file and call in code

Following is my appium capability set to run a test
cap = new DesiredCapabilities();
cap.setCapability(CapabilityType.PLATFORM, "Android");
cap.setCapability(CapabilityType.VERSION, "5.1.0");
cap.setCapability("deviceName", "mygeny510");
cap.setCapability("appPackage", "com.android.dialer");
cap.setCapability("appActivity", "com.android.dialer.DialtactsActivity");
driver = new AndroidDriver<MobileElement>(new URL("http://127.0.0.1:4723/wd/hub"), cap);
I want to keep the capabilities in a apm.json file
[
{
"platformName": "android",
"appPackage":"com.android.dialer",
"appActivity": "com.android.dialer.DialtactsActivity",
"deviceName": "mygeny510"
}
]
Now can anyone help to call the apm.json into the code instead writing each capabilities by using cap.setcapability(,)
You can place all the desired capabilities on to a seperate file and load the file in an other file for referencing it.
For eg,
I have the desired capabilities in env.rb
def abc
{
caps:
{
platformName: "iOS",
deviceName: "",
udid: "",
app: (File.join(File.dirname(__FILE__), "")),
bundleId: "",
automationName: "XCUITest",
xcodeOrgId: "",
xcodeSigningId: "",
platformVersion: "9.3.2",
noReset: "true",
fullReset: "false",
showIOSLog: "true"
}
}
end
Now you can go to the file where you want to launch this desired capabilities. For this, you would need to load the .json file into this file. I have used require_relative to load the file in order to call the method. Once you do that, you can start the session with
def AnyName
Appium::Driver.new(abc) #Pass capabilities for appium inside the driver
Appium.promote_appium_methods Object #Makes all appium_lib methods accessible from steps
$driver.start_driver #Starts appium driver before the tests begin
end
Hope this helps!