I'm working on an IoT project using AWS IoT and Greengrass v2 and I'm trying to integrate the ShadowManager component to use local shadows but when I deploy it on my device, it return a fatal exception during the synchronization step
{greengrass-root}/logs/greengrass.log
2021-09-15T09:54:29.044Z [INFO] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Executing sync request. {Type=LocalUpdateSyncRequest, thing name=mydevice, shadow name=}
2021-09-15T09:54:29.082Z [WARN] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Received conflict when processing request. Retrying as a full sync. {thing name=mydevice, shadow name=}
software.amazon.awssdk.aws.greengrass.model.ConflictError: Missed update(s) from the cloud
at com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest.execute(LocalUpdateSyncRequest.java:142)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:136)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:134)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.syncLoop(SyncHandler.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-09-15T09:54:29.083Z [INFO] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Executing sync request. {Type=FullShadowSyncRequest, thing name=mydevice, shadow name=}
2021-09-15T09:54:29.357Z [ERROR] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest: Could not execute cloud shadow get request. {thing name=mydevice, shadow name=}
2021-09-15T09:54:29.358Z [ERROR] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Skipping sync request. {thing name=mydevice, shadow name=}
com.aws.greengrass.shadowmanager.exception.SkipSyncRequestException: software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException: null (Service: IotDataPlane, Status Code: 403, Request ID: 84d49520-0162-7416-61a4-9973ecd32dad, Extended Request ID: null)
at com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest.getCloudShadowDocument(FullShadowSyncRequest.java:479)
at com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest.execute(FullShadowSyncRequest.java:93)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:136)
at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:134)
at com.aws.greengrass.shadowmanager.sync.SyncHandler.syncLoop(SyncHandler.java:270)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException: null (Service: IotDataPlane, Status Code: 403, Request ID: 84d49520-0162-7416-61a4-9973ecd32dad, Extended Request ID: null)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.iotdataplane.DefaultIotDataPlaneClient.getThingShadow(DefaultIotDataPlaneClient.java:221)
at com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper.getThingShadow(IotDataPlaneClientWrapper.java:89)
at com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest.getCloudShadowDocument(FullShadowSyncRequest.java:458)
... 10 more
It seems like the ShadowManager has not the good access rights but I checked multiple time and I have well added the good policies:
iot:GetThingShadow
iot:UpdateThingShadow
iot:DeleteThingShadow
I've tested with classic shadow and named shadow but same result.
The configuration of my ShadowManager is:
{
"synchronize": {
"coreThing": {
"classic": true,
"namedShadow": ["mydevice"]
}
}
}
It's also important to know that it worked the first time I've deployed it but after several minutes and some update messages published it failed (without changing anything).
Someone could help me on this?
Thank you
I was having a similar issue but with a client device.
This device was previously added to an old ggc v1. I migrated it to the ggc v2 but it seems that some config remained on the old server and it was overwriting my shadow.
So while testing with the help of the MQTT test I would create a shadow, ggc v2 would detect a conflict and then it will do a full sync deleting the shadow I had sent.
Using a completely new device avoided this error. I am still not able to sync sending the data from the client but at least from the MQTT test I can.
It looks like you are getting a 403 during a full sync operation when it tries to get the cloud shadow from IoT Device Shadow service. This 403 indicates that the device doesn't have permission to get device shadows. These permissions are required on the core device's AWS IoT policy. To confirm the correct permissions, please see the documentation about the minimal AWS IoT policy for core devices.
I was having the same issue and going crazy triple-checking configuration and IAM policies. Michael's answer will lead you to the right place, but the important thing to realize is that operations through the ShadowManager use AWS IoT Policies rather than IAM policies through the TokenExchangeRole.
From Device authentication and authorization for AWS IoT Greengrass
AWS IoT policies define the set of operations allowed for AWS IoT devices. Specifically, they allow and deny access to AWS IoT Core and AWS IoT Greengrass data plane operations, such as publishing MQTT messages and retrieving device shadows.
So, interactions with Device Shadows through the ShadowManager are IoT data plane operations which are checked against IoT Policies associated with the certificate identifying your Greengrass Core device. The TokenExchangeRole is not used for operations over the IoT data plane, meaning you don't need an IAM policy with permissions for shadow operations.
The previously linked document explains how to update an IoT Policy.
Related
We are experiencing an issue with running TestCafe where it will consistently stall on an API call. The application outside of TestCafe will consistently run without issue. From what I can trace, it seems to be within the TestCafe Hammerhead proxy request-pipeline.
I can see the request is (pending) but we don't see the request actually reaching the API server logs. There is an auth token in the request and I can see that Hammerhead will manage the auth flow from the header prefix:
~~~TestCafe added this prefix to control the authorization flow~~~
We have other API calls with non-standard auth headers (x-ns-authorization). TestCafe/Hammerhead will not stall on these calls and there is no added prefix. With the request that stalls, we cannot modify the auth header.
These are the versions we're running:
Node 14.20.1
└─┬ testcafe#2.1.0
├── testcafe-hammerhead#28.1.0
Hammerhead becomes a black box (aside from source code) as I don't see a way to debug the proxy. Is there a way to debug the proxy server? Or is it possible to prevent TestCafe from controlling the auth flow?
Also to note: the full tests will run successfully about <10% of the time. So it's a bit random when the tests don't stall.
I have asp.net core application hosted on GCP App Engine. When I try to deploy the application it fails on last step:
Updating service [name] (this may take several minutes)... ...failed
ERROR: (gcloud.app.deploy) Error Response: [9] An internal error occurred while processing task /app-engine-flex/flex_await_healthy/flex_await_healthy>blablabla.wm.1
The exception stack trace show that service running in background couldn't find MySQL table (that table obviously exists).
my app.yaml file:
service: XXX
runtime: custom
env: flex
automatic_scaling:
max_concurrent_requests: 80
min_num_instances: 1
max_num_instances: 1
resources:
cpu: XXX
memory_gb: XXX
beta_settings:
cloud_sql_instances: "XXX:XXXX:XXXX=tcp:3306"
It looks like the application is deployed properly despite the error. This is the only error and backgroud service desn't throw any exceptions at later point. In fact it works properly and can connect to the database.
My guess was that maybe GCP is checking health while the application is not connected do database. So I tried to add liveness_check and readiness_check to app.yaml and configured dedicated /healthcheck endpoint in my application but it didn't make any change.
Any ideas how to fix it and what might be a cause?
Deploying app with new version fixed the issue
I am trying to use UploadObjectExample.java code to upload a file to OCI object storage. I am running into connection timeout error while connecting to the object storage URL. The same config file is used by OCI CLI to successfully upload files to OCI config.
Here is the Error log:
Exception in thread "main" com.oracle.bmc.model.BmcException: (-1, null, true) Timed out while communicating to: https://objectstorage.us-ashburn-1.oraclecloud.com (outbound opc-request-id: 1EB5AA4A7FD64D58A54F876AD0C9E83B)
at com.oracle.bmc.http.internal.RestClient.convertToBmcException(RestClient.java:572)
at com.oracle.bmc.http.internal.RestClient.put(RestClient.java:380)
at com.oracle.bmc.objectstorage.ObjectStorageClient.putObject(ObjectStorageClient.java:1053)
at com.oracle.bmc.objectstorage.transfer.internal.SimpleRetry$1.apply(SimpleRetry.java:34)
at com.oracle.bmc.objectstorage.transfer.internal.SimpleRetry$1.apply(SimpleRetry.java:26)
at com.oracle.bmc.objectstorage.transfer.UploadManager.singleUpload(UploadManager.java:111)
at com.oracle.bmc.objectstorage.transfer.UploadManager.upload(UploadManager.java:73)
at UploadObjectExample.main(UploadObjectExample.java:74)
Caused by: javax.ws.rs.ProcessingException: java.net.SocketTimeoutException: connect timed out
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:284)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278)
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753)
at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
at org.glassfish.jersey.internal.Errors.process(Errors.java:229)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:752)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:445)
at org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:334)
at com.oracle.bmc.http.internal.ForwardingInvocationBuilder.put(ForwardingInvocationBuilder.java:141)
at com.oracle.bmc.http.internal.RestClient.put(RestClient.java:377)
Please test curl -v https://objectstorage.us-ashburn-1.oraclecloud.com from the same machine where the Java client times out, just to make sure there are no connection issues. If it works fine you may try to change the timeout value in ClientConfiguration. You can see more details here: https://github.com/oracle/oci-java-sdk/issues/92
Before creating a support ticket you might also try to create a new issue on github/oci-java-sdk.
without knowing more about the config file (I do not suggest you post it here), your home region and other elements it is very hard to help.
I would suggest you open a support ticket at https://support.oracle.com, making sure that you select the Cloud tab and the Service as "Oracle Cloud Infrastructure".
Are you using a proxy? If so, you may need to use the OCI Java SDK ApacheConnector.
This was an issue with the proxy. This was resolved by using the ash7 proxy.
I tried creating a new kube cluster via googleapis with oAuth authentication. But I am getting an error that
"HTTP Load Balancing requires the 'https://www.googleapis.com/auth/compute' scope.".
I came to know that google has updated the kube version to 1.2 the previous night in their console (until which I was able to create cluster using same method in v1.0)
I tried creating one via API explorer using google's oAuth, but it failed with same error.
I think the authscope has been updated, but I could not find the new authscope in any of 'google cloud platform container engine doc' or 'kubernetes latest release doc'. Can someone please help me in identifying the new authscope?
That error message was due to an error on our part while rolling out support for Kubernetes 1.2 in Google Container Engine. We've fixed the issues, and you can now create a container cluster using the api explorer. Sorry for the trouble.
That error message is referring to the scopes provided in the NodeConfig of the CreateCluster request. In 1.2, the "compute" scope is required to run the HTTP Load Balancer addon:
"nodeConfig": {
"oauthScopes": [
"https://www.googleapis.com/auth/compute"
]
}
If you don't want to add the https://www.googleapis.com/auth/compute scope to your nodes, you can also disable HTTP Load Balancing by passing in an AddonsConfig that disables it:
"addonsConfig": {
"httpLoadBalancing": {
"disabled": true
}
}
I am trying to deploy application from Jenkins build from Dev#cloud to AWS
using the instructions given at
https://developer.cloudbees.com/bin/view/DEV/ElasticBeanstalk
However, I am stuck because "cloudbees-deployer:elastic-beanstalk" is not
able to locate my application at AWS.
Here is the Console output from Jenkins Build
[cloudbees-deployer:elastic-beanstalk] Checking if S3 bucket
'photoid-reports-aws' exists...
[cloudbees-deployer:elastic-beanstalk] Checking if S3 bucket
'photoid-reports-aws' location...
[cloudbees-deployer:elastic-beanstalk] S3 bucket 'photoid-reports-aws'
location matches: us-east-1
[cloudbees-deployer:elastic-beanstalk] Uploading application to S3
bucket 'photoid-reports-aws/jenkins-photoid-reports-aws-9'...
[cloudbees-deployer:elastic-beanstalk] Application uploaded to S3
bucket 'photoid-reports-aws' with key
'jenkins-photoid-reports-aws-9/deploytest', version id 'null' and eTag
'427d78c1e5bfbaa7a1d10f46280236cc-8'
[cloudbees-deployer:elastic-beanstalk] Checking if application version
'prod-build' exists...
[cloudbees-deployer:elastic-beanstalk] Creating application version
'prod-build'...
com.cloudbees.plugins.deployer.exceptions.DeployException: No
Application named 'deploytest' found. (Service: AWSElasticBeanstalk;
Status Code: 400; Error Code: InvalidParameterValue; Request ID:
0cc70036-470e-11e4-90e5-1717b7862a74)
at com.cloudbees.plugins.deployer.engines.Engine.process(Engine.java:185)
at com.cloudbees.plugins.deployer.engines.Engine.perform(Engine.java:119)
at com.cloudbees.plugins.deployer.DeployBuilder.perform(DeployBuilder.java:104)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:825)
at hudson.model.Build$BuildExecution.build(Build.java:199)
at hudson.model.Build$BuildExecution.doRun(Build.java:160)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:606)
at hudson.model.Run.execute(Run.java:1684)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:232)
Caused by: com.amazonaws.AmazonServiceException: No Application named
'deploytest' found. (Service: AWSElasticBeanstalk; Status Code: 400;
Error Code: InvalidParameterValue; Request ID:
0cc70036-470e-11e4-90e5-1717b7862a74)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:820)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:439)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:245)
at com.amazonaws.services.elasticbeanstalk.AWSElasticBeanstalkClient.invoke(AWSElasticBeanstalkClient.java:1679)
at com.amazonaws.services.elasticbeanstalk.AWSElasticBeanstalkClient.createApplicationVersion(AWSElasticBeanstalkClient.java:540)
at com.cloudbees.plugins.deployer.impl.amazon.EngineImpl$DeployFileCallable.invoke(EngineImpl.java:355)
at com.cloudbees.plugins.deployer.impl.amazon.EngineImpl$DeployFileCallable.invoke(EngineImpl.java:224)
at com.cloudbees.plugins.deployer.engines.Engine$FingerprintingWrapper.invoke(Engine.java:271)
at com.cloudbees.plugins.deployer.engines.Engine$FingerprintingWrapper.invoke(Engine.java:259)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2462)
at hudson.remoting.UserRequest.perform(UserRequest.java:118)
at hudson.remoting.UserRequest.perform(UserRequest.java:48)
at hudson.remoting.Request$2.run(Request.java:328)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Build step 'Deploy applications' marked build as failure
Finished: FAILURE
Interesting. It looks like Cloudbees is assuming that you already have an application named "deploytest". The log looks like it is only trying to create a new application version as you can see after the S3 upload succeeded. It checks to make sure the app-version doesn't exist and then tries to create it.
What happens if you go through the Elastic Beanstalk Console to setup a new application with the name 'deploytest'? Just select the desired Environment Tier, Platform, and then Environment Type and try running that again. When it asks for application version, you can just use the sample app which should be selected by default.
Let me know if that helps.