I tried to see some questions like for example: How do I add a new dag to a running airflow service? but I did not succeed!
I uploaded the airflow in a virtual machine by following the steps of that link: https://medium.com/grensesnittet/install-apache-airflow-on-a-google-cloud-platform-virtual-machine-f9a5b01b6c33
And now I need to add new dags that I have created but I do not know how to find them! Could someone help me how to do this?
airflow.cfg:
dags_folder = /srv/airflow/dags
vm:
root#vm-airflow:/srv/airflow# ls
airflow-webserver.pid airflow.cfg airflow.db bin include lib logs unittests.cfg webserver.pid webserver_config.py
Related
Good afternoon friends, I need to disable "tcp_timestamps" in ebs for a security test. I created an environment for the test, my ".ebextesions" doesn't work. I looked at the aws doc, I created a directory with the name my-app, one with the name .ebextesions and then the file "disable_tcp_timestamps" with the configuration content.
It didn't work the tcp is still active, can you help me?
I used this config
commands:
01_disable_timestamp:
command: echo "0" > /proc/sys/net/ipv4/tcp_timestamps
I've developped an API with Node.Js, Express, Prisma and Mysql in local firstly. After that it works, I have deployed my API on Heroku and I took the ClearDB add-on to have a Mysql DB on Heroku.
So the deployment is OKAY when I go on my root root URI I have the "Cannot GET /" message, and when I try to connect to my ClearDB with MysqlWorkbench I have my tables, columns etc...
The main problem is from Prisma.
When I go to the "Run console" of my Heroku's project, the command npx prisma init works perfectly BUT when I type npx prisma migrate deploy || dev or also if I try to npx prisma db push I have this error =>
Error: Get Config: Schema parsing - Error while interacting with query-engine-node-api library
Error code: P1012
error: Environment variable not found: DATABASE_URL.
--> schema.prisma:10
|
9 | provider = "mysql"
10 | url = env("DATABASE_URL")
|
All my code is in a GitHub repo, I've configured my .env (which is in the root folder of my server) like this :
DATABASE_URL="mysql://<username>:<my-password>#eu-cdbr-west-30.cleardb.net/heroku_36d295ebb6686a2"
NODE_ENV="development"
APP_SECRET="jwtsecret12"
NODE_PATH="./src"
ACCESS_TOKEN_SECRET="651651651848754cdfce9fz8ef4ef54se8f4sef48s69ef84e"
I hope you have all the informations that you need to help me :)
PS : Locally my project works perfectly
Waiting for your answers, thank you very much !
Your .env file is irrelevant. It should not be used on Heroku (and should not be tracked in your repository).
ClearDB provides an environment variable called CLEARDB_DATABASE_URL, not DATABASE_URL. You can either change your code to use this variable instead of DATABASE_URL, or you can set DATABASE_URL to the same value:
Retrieve your database URL by issuing the following command:
heroku config | grep CLEARDB_DATABASE_URL
CLEARDB_DATABASE_URL => mysql://adffdadf2341:adf4234#us-cdbr-east.cleardb.com/heroku_db?reconnect=true
Copy the value of the CLEARDB_DATABASE_URL config variable.
If you’re using Ruby on Rails and the mysql2 gem, you will need to change the mysql:// scheme in the CLEARDB_DATABASE_URL to mysql2://
heroku config:set DATABASE_URL='mysql://adffdadf2341:adf4234#us-cdbr-east.cleardb.com/heroku_db?reconnect=true'
Adding config vars:
DATABASE_URL => mysql2://adffd...b?reconnect=true
Restarting app... done, v61.
The connection information for Heroku Postgres can change at any time, but since the ClearDB documentation provides the preceding guidance I would hope that it does not do so.
This is the error :
reverie-pc#reveriepc-Latitude-3400:~/VasanthkumarV/prisma$ sudo npm install -g prisma
[sudo] password for reverie-pc:
npm WARN deprecated request#2.88.2: request has been deprecated, see
https://github.com/request/request/issues/3142
/usr/bin/prisma -> /usr/lib/node_modules/prisma/dist/index.js
+ prisma#1.34.10
updated 1 package in 29.734s
(base) reverie-pc#reveriepc-Latitude-3400:~/VasanthkumarV/prisma$ prisma init test
? Set up a new Prisma server or deploy to an existing server? Use existing database
? What kind of database do you want to deploy to? MySQL
? Does your database contain existing data? Yes
? Enter database host localhost
? Enter database port 3306
? Enter database user root
? Enter database password [hidden]
? Please select the schema you want to introspect database_test
Introspecting database database_test 435ms
Created datamodel definition based on 24 tables.
? Select the programming language for the generated Prisma client Prisma JavaScript Client
Created 3 new files:
prisma.yml Prisma service definition
datamodel.prisma GraphQL SDL-based datamodel (foundation for database)
docker-compose.yml Docker configuration file
Next steps:
1. Open folder: cd test
2. Start your Prisma server: docker-compose up -d
3. Deploy your Prisma service: prisma deploy
4. Read more about introspection:url
▸ Syntax Error: Expected Name, found Int "1"
Get in touch if you need help: https://slack.prisma.io/
To get more detailed output, run $ export DEBUG="*"
(node:14055) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated
Generating schema... !
How to resolve this error..and what is the procedure to connect Prisma server with local database (MySQL)?? and what about the prisma deployment??
How to connect prisma with existing db?
It looks like you are using Prisma 1 which is currently in maintenance mode.
Given that this looks like a new project, I'd suggest you take a look at Prisma 2 which includes many improvements and a simpler mental model.
I am creating an EMR cluster, and using jupyter notebook to run some spark tasks.
My tasks die after approximately 1 hour of execution, and the error is:
An error was encountered:
Invalid status code '400' from https://xxx.xx.x.xxx:18888/sessions/0/statements/20 with error payload: "requirement failed: Session isn't active."
My understanding is that it is related to the Livy config livy.server.session.timeout, but I don't know how I can set it in the bootstrap of the cluster (I need to do it in the bootstrap because the cluster is created with no ssh access)
Thanks a lot in advance
On EMR, livy-conf is the classification for the properties for livy's livy.conf file, so when creating an EMR cluster, choose advanced options with Livy as an application chosen to install, please pass this EMR configuration in the Enter Configuration field.
[{'classification': 'livy-conf','Properties': {'livy.server.session.timeout':'5h'}}]
On EMR, Livy binary is located at /etc/livy/, and so the config file is at /etc/livy/conf/livy.conf
To verify this,
Create an EMR cluster with a known ec2 key-pair, Livy and above config
Using the ec2 key-pair, login to the EC2 Master node associated with the cluster ssh -i some-ec2-key-pair.pem hadoop#ec2-00-00-00-0.ca-region-n.compute.amazonaws.com
Navigate to /etc/livy/conf, vim livy.conf & see the updated value of livy.server.session.timeout
If you don't want the Livy session to go down at all, then set the property livy.server.session.timeout-check to false in /etc/livy/conf/livy.conf.
Another way to do that if you don’t want to recreate the cluster is:
go to /etc/livy/conf/livy.conf and set the livy.server.session.timeout property to the value you would like.
After that, run sudo restart livy-server to make the configuration applied.
Im trying to learn Hyperledger Composer but seems to be a relatively new technology, i mean there are few tutorials and few solutions to a lot of questions, tutorial does not mention possible error case when following the commands and which means there are is also no solution for those errors.
I have joined the composer channel in their community chat, looks like its running in Discord or something, and asked the same question without a response, i have a better experience here in SO.
This is the problem: I have deployed my business network, installed it, started it, created my network admin card and imported it, then to test if everything is ok i have to command composer network ping --card NAME-OF-MY-ADMIN-CARD
And this error comes:
juan#JuanDeDios:~/proyectos/inovacion/a3-poliza-microservice$ composer network ping --card admin#a3-policy-microservice
Error: transaction returned with failure: AccessException: Participant 'org.hyperledger.composer.system.NetworkAdmin#admin' does not have 'READ' access to resource 'org.hyperledger.composer.system.Network#a3-policy-microservice#0.0.1'
Command failed
I think that it has to do something with the permission.acl file, and gave permission to everyone to everything so there would not be any restrictions to anyone, and tryied again, but failed.
So i thought i had to uninstall my business network and create it again, i deleted my .bna and my network.card files also so everything would be created again, but the same error result.
My other attempt was to update the business network, but didn't work, the same error happened and I'm sure i didn't miss any step from the tutorial. I do also followed the playground tutorial. What i have not done its to create another app with the Yeoman but i will do if i don't find a solution to this problem which would not require me to create another app.
This were my steps:
1-. Created my app with Yeoman
yo hyperledger-composer:businessnetwork
2-. Selected Apache-2.0 for my license
3-. Created a3-policy-microservice as the name of the business network
4-. Created org.microservice.policy (Yeah i switched names but Im totally aware)
5-. Generated my app with a template selecting the NO option
6-. Created my assets, participants and transactions
7-. Changed my permission rules to mine
8-. I generated the .bna file
composer archive create -t dir -n .
9-. Then installed my bna file
composer network install --card PeerAdmin#hlfv1 --archiveFile a3-policy-microservice#0.0.1.bna
10-. Then started my network and created my networkadmin card
composer network start --networkName a3-policy-network --networkVersion 0.0.1 --networkAdmin admin --networkAdminEnrollSecret adminpw --card PeerAdmin#hlfv1 --file networkadmin.card
11-. Imported my card
composer card import --file networkadmin.card
12-. Tried to ping my network
composer network ping --card admin#a3-poliza-microservice
And the error happens
Later i tried to create everything again shutting down my fabric and started it again and creating the network from the first step.
My other attempt was to change the permissions and upgrade my bna network, but it failed too. Im running out of options
Hope this description its not too long to ignore it. Thanks in advance
thanks for the question!
First possibility is that your network name is a3-policy-network but you're pinging a network called a3-poliza-microservice - once you do get the correct ACLs in place (currently, that's the error you're trying to resolve).
The procedure for upgrade would normally be the procedure below:
After your step 12 (where you can't ping the business network due to restrictive ACL conditions, assuming you are using the right network name) you would have:
Make the changes to to include your System ACLs this time eg.
/**
* Sample access control list.
*/
rule SystemACL {
description: "System ACL to permit all access"
participant: "org.hyperledger.composer.system.Participant"
operation: ALL
resource: "org.hyperledger.composer.system.**"
action: ALLOW
}
rule NetworkAdminUser {
description: "Grant business network administrators full access to user resources"
participant: "org.hyperledger.composer.system.NetworkAdmin"
operation: ALL
resource: "**"
action: ALLOW
}
rule NetworkAdminSystem {
description: "Grant business network administrators full access to system resources"
participant: "org.hyperledger.composer.system.NetworkAdmin"
operation: ALL
resource: "org.hyperledger.composer.system.**"
action: ALLOW
}
Update the "version" field in your existing package.json in your Business Network project directory (ie need to change it next increment - eg. update the version property from 0.0.1 to 0.0.2.)
From the same directory, run the following command:
composer archive create --sourceType dir --sourceName . -a a3-policy-network#0.0.2.bna
Now install the new business network code firstly:
composer network install --card PeerAdmin#hlfv1 --archiveFile a3-policy-network#0.0.2.bna
Then perform the requisite upgrade step (single '-' for short form of the parameter):
composer network upgrade -c PeerAdmin#hlfv1 -n a3-policy-network -V 0.0.2
After a few seconds, ping the network again to see ACL changes are now in effect:
composer network ping -c a3-policy-network