Packer SSH timeout when setting custom VPC, subnet and security group - packer

Question

So I needed to be able to move my packer builders inside a private VPC and add a locked down security group that only allowed ssh from a restricted range of IPs, thus:
"builders": [{
"type": "amazon-ebs",
"associate_public_ip_address": false,
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "{{user `aws_region`}}",
"source_ami_filter": {
"filters": {
"virtualization-type": "hvm",
"name": "{{user `ami_source_name`}}",
"root-device-type": "ebs"
},
"owners": ["{{user `ami_source_owner_id`}}"],
"most_recent": true
},
"instance_type": "t3.small",
"iam_instance_profile": "{{user `iam_instance_profile`}}",
"ssh_username": "{{user `ssh_username`}}",
"ami_name": "{{user `ami_name_prefix`}}_{{user `ami_creation_date`}}",
"ami_users": "{{user `share_amis_with_account`}}",
"ebs_optimized": true,
"vpc_id": "vpc-123456",
"subnet_id": "subnet-123456",
"security_group_id": "sg-123456",
"user_data_file": "scripts/disable_tty.sh",
"launch_block_device_mappings": [{
"device_name": "{{user `root_device_name`}}",
"volume_size": 10,
"volume_type": "gp2",
"delete_on_termination": true
}],
"tags": {
"packer": "true",
"ansible_role": "{{user `ansible_role`}}",
"builtby": "{{user `builtby`}}",
"ami_name": "{{user `ami_name_prefix`}}_{{user `ami_creation_date`}}",
"ami_name_prefix": "{{user `ami_name_prefix`}}",
"project": "{{user `project`}}"
}
}]
To start with I added "associate_public_ip_address:false" (false is the default as well) as every time I ran packer the host was assigned a public ip address but even adding that it still picks up a public ip????????
I used a security group that I had assigned to Jenkins build slaves which also communicate over port 22 and I haven't had any issue with accessing them from any part of my infrastructure.
I get this error:
1562344256,,ui,error,Build 'amazon-ebs' errored: Timeout waiting for SSH.
1562344256,,error-count,1
1562344256,,ui,error,\n==> Some builds didn't complete successfully and had errors:
1562344256,amazon-ebs,error,Timeout waiting for SSH.
1562344256,,ui,error,--> amazon-ebs: Timeout waiting for SSH.
During the wait period for SSH to respond I was able to nc -v 1.2.3.5 22 and I get a connection so the security group is allowing communications on port 22 from my IP address.
If I change the security group to 0.0.0.0/0 it connects straight away but why when I can nc to port 22 with the restricted security group can packer not initiate an SSH connection? Is packer trying to use the public IP address that I can not for the life of me turn off?
I thought it might be quite helpful to tcpdump the traffic on port 22 to see what was happening but I have a locked down laptop that does not allow the install of that particular handy item.
I can also ssh to the builder from my laptop but get a Too many authentication failures error and can't log in to see what is going on.

Answer 1

So the reason that the packer builder is getting a public ip is down to the subnet settings - map_public_ip_on_launch = true.
So answer is build a new private subnet for the packer builder, build a new NAT GW in the public subnet then route from the private subnet to the NAT GW with a new routing table.