Per the google docs (https://cloud.google.com/logging/docs/export/using_exported_logs#log_entries_in_google_bigquery) I have set up my GCP app engine to auto-export to big query. However, I am running nodejs using bunyan. My logs are in json format. I'd like to take advantage of the cloud logging "structPayload" LogEntry, but the auto-export seems to automatically dump it into a "textPayload". Is there any way to configure this?
I'm one of the engineers working on Cloud Logging. We haven't yet announced the structured logging feature, and documentation will be available when we do, but the functionality is present in the cloud logging plugin and can be used.
In your case, if you edit the configuration file that is capturing your logs (under /etc/google-fluentd/config.d/), configure 'format json', then 'service google-fluentd reload', you should see your logs ingested as structPayload - each json field will become a column in BigQuery.
See the tail input plugin documentation for more details on the configuration options: http://docs.fluentd.org/articles/in_tail
Related
Could it be possible for already created (Yarn/Hadoop) cluster to disable logging for all servers inside ?
I can't find anything like it. Is there anything in Dataproc or Compute Engine which can help me to disable the logs ?
One easy way would be to create an exclusion in Stackdriver Logging that would prevent logs from that cluster from being ingested into Stackdriver.
You can create a resuorce based exclusion in Stacdriver - select a DataProc cluster you want and it will stop collecting any logs - hence bill you for that.
Go to Logs Ingestion page, select Exclusions and click blue button "create exclusion".
As a resource type select "Cloud Dataproc Cluster" > your_cluster_name > All cluster_uuid as shown below. Also - select "no limit" for time frame.
Fill the "Name" field on the right and again click blue button "Create Exlusion".
You can create up to 50 exclusion queries in StackDriver.
With little help and suggestion from Google support, there is complete solution to skip logging for whole yarn/hadoop cluster.
That can be possible only when create new cluster from dataproc either by google cloud page or console.
Property which should to be set in cluster properties filed:
dataproc:dataproc.logging.stackdriver.enable to be false
More info at: https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/cluster-properties
If you create cluster through console you can referral https://cloud.google.com/sdk/gcloud/reference/dataproc/clusters/create#--properties and to use command like:
gcloud.cmd --properties 'dataproc:dataproc.logging.stackdriver.enable=false'
I'm using CasperJS for automated UI tests. I've done the basic UI testing and validation with some random data, kind of POC. I've set up this automation using bash script which kicks to start the web server, load MySQL data from SQL file, start CasperJS test cases, stop the web server, check the log files.
Now, I want to start the testing with some good known status of data which are stored in MySQL. So that I can test the list data and form data with detailed field information with some known database status. How should I know the status of data in the database at a moment?
1) Should I use pre-populated JSON dumped file which has status and details about all data?
2) Should I use web service API? (web service APIs are being used to show/save/delete data from the web page)
Let's take an example. I've 5 users in Users table. Now when I open the home page it shows 5 users with some rough details. When I click on any record from the list of users, it shows a form with detailed information about that user. The webpage is requesting to the web application to get the detail about a user with the help of user_id to show the detailed user data in a form. Now I want to check that all the data in that form is populated correctly. So at the next step, what would be the preferred way, should I read content from JSON dumped file or should I use web service API (like webpage does).
Searching this problem online, I also found MYSQL HTTP plugin. Should I consider this as well? and How safe it is to use? (I know from the docs that this plugin is not for the production, it is just for testing purpose only. :) )
For the main question in cases like this I would change the database connection string to your testing database (this is a clone).
In your case use your bash script to change the connection string (file copy?) automatically before you run the tests. And when completed change back.
Your testing database is a direct clone of your dev/live databsae but with ONLY the test data you want. Downside is you need to keep the schema in sync with DEV/LIVE.
Also another point to take into considertion is if your testing changes state (post). If so your testing data might be out of sync. One way is get around this is to drop foreign keys, truncate the data and load in a dump file.
HTH
When I try to load a big CSV from a zip file, the execution log give me the following error:
----------------------------------------- Error details ------------------------------------------
Component [Clientes:CLIENTES1] finished with status ERROR.
The size of data buffer is only 100663296. Set appropriate parameter in defaultProperties file.
--------------------------------------------------------------------------------------------------
How can I set the appropriate parameter in defaultProperties file?
I tried this link, but my cloudconnect run configurations page is different from the link:
I've created the parameters file and filled the additional parameters with the right values like said the tutorial (code bellow) and the same error appear in the screen.
Name: -config; Value: new_buffer_size.txt
The new_buffer_size.txt content have just this line:DEFAULT_INTERNAL_IO_BUFFER_SIZE = 200000000
How can I solve this problem? I need to solve this before the world explodes.
CloudConnect is designed to develop ETL(s), which can be run on GoodData cloud workers and therefore some lower level settings are surpassed as in this case. The only legitimate way is to modify the ETL the way it can process the data with current settings. Regarding to docs, the referenced article is outdated. GoodData docs team is aware if it and they are preparing docs refactoring.
Note: As you have probably noticed, CloudConnect is being powered by Javlin's Clover ETL, therefore feel free to check their forums, as you would find there how to overcome the issue on lower level (no UI), but it would work only for data processing on the local machine.
I have installed a standalone instance of wso2 API Manager 1.10.0 with the CARBON-PATCH-4.4.0-0084 installed. I am walking through the PhoneVerification tutorial and have published the API as instructed. I created a new user for the store and am trying to subscribe to the PhoneVerification-2.0.0 API but cannot because there is no DefaultApplication in the Applications dropdown list. It is missing from the list. I tried to add and application and get an error dialog saying Missing Parameters. I am stuck and cannot go any further. One additional piece of information, I am using mysql instead of the h2 default. I followed all the instructions to setup mysql and had no problems. Also I changed the admin password as well.
I have tried API Manager 1.10 with security patch CARBON-PATCH-4.4.0-0084 with no issues. I can create APIs and subscribe using the DefaultApplication. I tried with default database and a new user who is assigned to 'Internal/subscriber' role.
Please check if you have any other changes for Store Web app at 'wso2am-1.10.0/repository/deployment/server/jaggeryapps/store'. You have to replace 'store' and 'publisher' apps as instructed in the Readme of patch.
(iii) Merge and Replace resource/store to /repository/deployment/server/jaggeryapps/store
(iv) Merge and Replace resource/publisher to /repository/deployment/server/jaggeryapps/publisher
To check if the Mysql configuration worked properly, please check if there are any errors in Carbon log, at 'repository/logs/wso2carbon.log' file related to that. And double check if userstore and api-manager database tables are created properly and configured in 'repository/conf/datasources/master-datasources.xml' properly.
I need to get all configuration settings (current role or all roles not matter) from Azure cscfg file. I want to do this because i dont want to get all values one by one via RoleEnvironment.GetConfigurationSettingValue(key) method.
Is there any way to do this?
Regards
The short answer is 'no' the RoleEnvironment does not support getting all the configuration setting values.
A slightly longer answer is that getting configuration settings from the role environment in the current implementation is done through a call to native code. The separation of Windows Azure Application from Windows Azure Configuration and the ability to swap settings on a running application is at the root of this somehow. This is done inside of msshrtmi.dll (which should mean something like Microsoft Shared Runtime Managed Interop). This is the only reference Microsoft.WindowsAzure.ServiceRuntime.dll has apart from standard references to .NET.
Here is the method call to native code (I have not gone further than this):
[MethodImpl(MethodImplOptions.Unmanaged, MethodCodeType=MethodCodeType.Native), SuppressUnmanagedCodeSecurity, DllImport("", EntryPoint="", CallingConvention=CallingConvention.StdCall, SetLastError=true)]
internal static extern unsafe int modopt(IsLong) modopt(CallConvStdcall) RdGetApplicationConfigurationSetting(ushort modopt(IsConst), ushort*);
It might seem like a slightly round-about way of doing it, but if you want to get the configurations for all the roles in a deployment you can use the management api.
Kudu has an API for this.
You get to Kudu like via App Services > Advanced Details > Go
https://{app-service-name}.scm.azurewebsites.net/
or
https://{app-service-name}-{slot-name}.scm.azurewebsites.net/
The Url for the settings API is:
https://{app-service-name}.azurewebsites.net/api/settings or https://{app-service-name}-{slot-name}.scm.azurewebsites.net/