I've been searching for this for a while. I don't have access to the binary items used to build the image because an artifactory migration ruined the repo. There is one particularly precious binary I would love to extract from the image. I know docker save would save me, but I don't have access to docker, only to the oc client.
EDIT:
After looking around a little, thought that docker-registry API should be the way to go. Debugging oc client and logs of the docker-registry pods, found that both v1 and v2 API versions seem to be used.
Somehow cannot get any further than the version check.
Getting the auth token and registry url from oc:
TOKEN=`oc whoami -t`
URL="https://"`oc -n default get route docker-registry -o jsonpath="{.status.ingress[0].host}"
Then getting a correct response to:
curl -k -X GET -H "Authorization: Bearer $TOKEN" "$URL/v2/"
...
HTTP/1.1 200 OK
but:
curl -k -X GET -H "Authorization: Bearer $TOKEN" "$URL/v2/_catalog"
...
HTTP/1.1 400 Bad Request
You can log in to the internal image registry if exposed and then pull the image back down to your local system and do what you want with it. Instructions for logging in can be found in:
http://cookbook.openshift.org/image-registry-and-image-streams/how-do-i-push-an-image-to-the-internal-image-registry.html
That talks about doing a push, but you want to do a pull.
Related
I want to upload the new bluprint.json file to my ambari cluster as the following
curl -u admin:admin -H "X-Requested-By: ambari" -X GET http://10.14.5.40:8080/api/v1/clusters/HDP6?format=blueprint -o /tmp/1-HDP6_blueprint.json
when I run it , seems that every thing is ok because we not get any warning /error
but when I read the ambari GUI parameters I see that the new bluprint.json not affected the ambari cluster with the new configuration
how to debug this ?, or how to get notification from the curl ... syntax about what happens ?
Please note that curl command you used is for downloading existing cluster configuration in blueprint format (its XGET).
You will have to use curl -XPOST to register and upload new blueprint to ambari.
curl --verbose -H "X-Requested-By: ambari" -X POST -u admin:admin http://10.14.5.40:8080/api/v1/blueprints/:HDP6new?validate_topology=false --data "#./blueprint.json"
Also note that, to change existing cluster configuration, uploading modified bluetooth is not correct way. You may refer this document for modifying configurations.
I am following this guide on Github and I am not able run the example mapreduced job mentioned in Step 5.
I am aware that this file no longer exists:
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar
And I am aware that the same file can now be found here:
/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar
So I form my call as below:
curl -v -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/$user/jobs" -d '{"jar":"/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar","class_name":"WordCount","lib_jars":"/usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar","input":"testdir","output":"testoutput"}' -H "Content-Type: application/json" -H "X-Auth-Token: $TOKEN"
The input directory exists in my hdfs user space and there is a file called testdata.txt inside it. The testoutput folder does not exist in my hdfs user space since I know it creates problems.
When I execute this curl command, the error I get is {"success":"false","error":1} which is not very descriptive. Is there something I am missing here?
This has been just tested with my user frb and a valid token for that user:
$ curl -X POST "http://computing.cosmos.lab.fiware.org:12000/tidoop/v1/user/frb/jobs" -d '{"jar":"/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar","class_name":"wordcount","lib_jars":"/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar","input":"testdir","output":"outputtest"}' -H "Content-Type: application/json" -H "X-Auth-Token: xxxxxxxxxxxxxxxxxxx"
{"success":"true","job_id": "job_1460639183882_0011"}
Please observe the fat jar with the MapReduce examples in the "new" cluster (computing.cosmos.lab.fiware.org) is at /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar, as detailed in the documentation. /usr/lib/hadoop-0.20/hadoop-examples-0.20.2-cdh3u6.jar was the fat jar in the "old" cluster (cosmos.lab.fiware.org).
EDIT 1
Finally, the user had no account in the "new" pair of clusters of Cosmos in FIWARE LAB (storage.cosmos.lab.fiware.org and computing.cosmos.lab.fiware.org), where Tidoop runs, but in another "old" cluster (cosmos.lab.fiwre.org). Thus, the issue was fixed by simply provisioning an account in the "new" ones.
Using WebHDFS API of Cosmos generates an expired certificate response.
Using this url: https://cosmos.lab.fi-ware.org:13000/
we can see certificate seems expired
Do we need updated certificate or any way to go around this problem?
The certificate must be renewed, for sure. In the meantime, you can simply ignore the certificate. If you are using curl, use the -k option:
$ curl -k -X POST "https://cosmos.lab.fiware.org:13000/cosmos-auth/v1/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=password&username=frb#tid.es&password=MY_PASSWORD_IS_PRIVATE"
I've recently seen the access to Cosmos' WebHDFS in FIWARE Lab has been protected with OAuth2. I know I have to add a OAuth2 token to the request in order to continue using WebHDFS, but:
How can I get the token?
How the token is added to the request?
Without the token, the API always returns:
$ curl -X GET "http://cosmos.lab.fi-ware.org:14000/webhdfs/v1/user/gtorodelvalle?op=liststatus&user.name=gtorodelvalle"
Auth-token not found in request header
Yes, now WebHDFS access is protected with OAuth2. This is part of the general mechanism for pretecting REST APIs in FIWARE, which performs authentication and authorization. You can find more details here.
First of all, you must request an OAuth2 token to the Cosmos tokens generator. This is a service running in cosmos.lab.fiware.org:13000. You can do this using any REST client, the easiest way is using the curl command:
$ curl -k -X POST "https://cosmos.lab.fiware.org:13000/cosmos-auth/v1/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=password&username=frb#tid.es&password=xxxxxxxx"
{"access_token": "qjHPUcnW6leYAqr3Xw34DWLQlja0Ix", "token_type": "Bearer", "expires_in": 3600, "refresh_token": "V2Wlk7aFCnElKlW9BOmRzGhBtqgR2z"}
As you can see, your FIWARE Lab credentials are required in the payload, in the form of a password-based grant type.
Once the access token is got (in the example above, it is qjHPUcnW6leYAqr3Xw34DWLQlja0Ix), simply add it to the same WebHDFS request you were performing in the past. The token is added by using the X-Auth-Token header:
$ curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/frb/path/to/the/data?op=liststatus&user.name=frb" -H "X-Auth-Token: qjHPUcnW6leYAqr3Xw34DWLQlja0Ix"
{"FileStatuses":{"FileStatus":[...]}}
If you try the above request with a random token the server will return the token is not valid; that's because you have not authenticated properly:
$ curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/frb/path/tp/the/data?op=liststatus&user.name=frb" -H "X-Auth-Token: randomtoken93487345"
User token not authorized
The same way, if using a valid token but trying to access another HDFS userspace, you will get the same answer; that's because you are not authorized to access any HDFS userspace but the one owned by you:
$ curl -X GET "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/fgalan/path/tp/the/data?op=liststatus&user.name=fgalan" -H "X-Auth-Token: qjHPUcnW6leYAqr3Xw34DWLQlja0Ix"
User token not authorized
IMPORTANT UPDATE:
From summer 2016, cosmos.lab.fiware.org is not workin anymore. Instead, a pair of clusters, storage.cosmos.lab.fiware.org and computing.cosmos.lab.fiware.org have been setup. Regarding the auth server of Cosmos, it currently run in computing.cosmos.lab.fiware.org, port TCP/13000.
The right request must be:
curl -X POST "https://cosmos.lab.fi-ware.org:13000/cosmos-auth/v1/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=password&username=user#domain.com&password=yourpassword" -k
The url was incorrect, the correct is https://cosmos.lab.fi-ware.org:13000
-k is for turn off certificate verification
I'm trying to use wget 1.11.4 running on Windows Server 2003 to do some JSON with a web site. I can retrieve info OK but I can't send any. I'm not positive that my JSON is formatted correctly but there seems to be a more fundamental problem.
If I do:
wget --header="Authorization: Bearer <redacted>" --no-check-certificate --post-data="{'workspaces':{'title':'CMC Regulatory Support','creator_role':'maven'}}" https://api.mavenlink.com/api/v1/workspaces.json
I get a JSON error back from the site "ERROR 422: Unprocessable Entity." But if I add another --header:
wget -S --header="Authorization: Bearer <redacted>" -–header="Content-Type: application/json" --no-check-certificate --post-data="{'workspaces':{'title':'CMC Regulatory Support','creator_role':'maven'}}" https://api.mavenlink.com/api/v1/workspaces.json
I get:
SYSTEM_WGETRC = c:/progra~1/wget/etc/wgetrc
syswgetrc = C:\Program Files\GnuWin32/etc/wgetrc
wget: invalid option -- û
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
but the manual clearly states that multiple --header switches are allowed, and I see that swithc used in examples on the web?. WTF?
Your second --header parameter before the Content-Type contains different dashes.
-–header="Content-Type
If you fix it wget will send the request.