Abort: unknown largefiles usercache location - mercurial

I've got older Debian Jessie server with Mercurial 3.1.2 and Phabricator. Long time Phabricator protest that HG is old and there is secure problems and I decide to make brand new VM with latest HG.
Old:
lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 8.11 (jessie)
Release: 8.11
Codename: jessie
hg version
Mercurial Distributed SCM (version 3.1.2)
(see http://mercurial.selenic.com for more information)
New:
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
hg version
Mercurial Distributed SCM (version 4.8.2)
(see https://mercurial-scm.org for more information)
Copyright (C) 2005-2018 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Enabled extensions:
largefiles internal
and configured:
hg debuginstall
checking encoding (UTF-8)...
checking Python executable (/usr/bin/python2)
checking Python version (2.7.16)
checking Python lib (/usr/lib/python2.7)...
checking Python security support (sni,tls1.0,tls1.1,tls1.2)
checking Mercurial version (4.8.2)
checking Mercurial custom build ()
checking module policy (c)
checking installed modules (/usr/lib/python2.7/dist-packages/mercurial)...
checking registered compression engines (bz2, bz2truncated, none, zlib, zstd)
checking available compression engines (bz2, bz2truncated, none, zlib, zstd)
checking available compression engines for wire protocol (zstd, zlib, bz2, none)
checking "re2" regexp engine (missing)
checking templates (/usr/share/mercurial/templates)...
checking default template (/usr/share/mercurial/templates/map-cmdline.default)
checking commit editor... (sensible-editor)
checking username (myuser <myuser#gmail.com>)
no problems detected
hg --debug showconfig
starting pager for command 'config'
read config from: /usr/etc/mercurial/hgrc
read config from: /etc/mercurial/hgrc
read config from: /etc/mercurial/hgrc.d/cacerts.rc
read config from: /etc/mercurial/hgrc.d/hgext.rc
read config from: /root/.hgrc
read config from: /root/.config/hg/hgrc
/etc/mercurial/hgrc:9: extensions.largefiles=
/etc/mercurial/hgrc:19: trusted.users=root, topperl, www-data
/etc/mercurial/hgrc:12: ui.username=myuser <myuser#gmail.com>
--verbose: ui.verbose=False
--debug: ui.debug=True
--quiet: ui.quiet=False
pager: ui.formatted=True
pager: ui.interactive=False
/etc/mercurial/hgrc.d/cacerts.rc:4: web.cacerts=/etc/ssl/certs/ca-certificates.crt
Rsynced all the repositories and setup hgweb.wsgi to act like a web server (Apache2 vhost as it's easy for me)
# An example WSGI for use with mod_wsgi, edit as necessary
# See https://mercurial-scm.org/wiki/modwsgi for more information
# Path to repo or hgweb config to serve (see 'hg help hgweb')
config = "/var/www/vhost/hg/cgi-bin/hgweb.config"
# Uncomment and adjust if Mercurial is not installed system-wide
# (consult "installed modules" path from 'hg debuginstall'):
#import sys; sys.path.insert(0, "/path/to/python/lib")
# Uncomment to send python tracebacks to the browser if an error occurs:
import cgitb; cgitb.enable()
# enable demandloading to reduce startup time
from mercurial import demandimport; demandimport.enable()
from mercurial.hgweb.hgwebdir_mod import hgwebdir
import os
os.environ["HGENCODING"] = "UTF-8"
from mercurial.hgweb import hgweb
application = hgweb(config)
Python version is:
Python 2.7.16 (default, Apr 6 2019, 01:42:57)
[GCC 8.3.0] on linux2
When push large files I receive error:
mod_wsgi (pid=3945): Exception occurred processing WSGI script '/var/www/vhost/hg/cgi-bin/hgwebdir.w
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/hgwebdir_mod.py", line 352, in run_wsgi
for r in self._runwsgi(req, res):
File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/hgweb_mod.py", line 307, in run_wsgi
for r in self._runwsgi(req, res, repo):
File "/usr/lib/python2.7/dist-packages/mercurial/hgweb/hgweb_mod.py", line 333, in _runwsgi
rctx, req, res, self.check_perm)
File "/usr/lib/python2.7/dist-packages/mercurial/wireprotoserver.py", line 221, in handlewsgireque
_callhttp(repo, req, res, proto, cmd)
File "/usr/lib/python2.7/dist-packages/mercurial/wireprotoserver.py", line 436, in _callhttp
rsp = wireprotov1server.dispatch(repo, proto, cmd)
File "/usr/lib/python2.7/dist-packages/mercurial/wireprotov1server.py", line 74, in dispatch
return func(repo, proto, *args)
File "/usr/lib/python2.7/dist-packages/mercurial/wireprotov1server.py", line 196, in batch
result = func(repo, proto, *[data[k] for k in keys])
File "/usr/lib/python2.7/dist-packages/hgext/largefiles/proto.py", line 89, in statlfile
filename = lfutil.findfile(repo, sha)
File "/usr/lib/python2.7/dist-packages/hgext/largefiles/lfutil.py", line 111, in findfile
elif inusercache(repo.ui, hash):
File "/usr/lib/python2.7/dist-packages/hgext/largefiles/lfutil.py", line 100, in inusercache
path = usercachepath(ui, hash)
File "/usr/lib/python2.7/dist-packages/hgext/largefiles/lfutil.py", line 71, in usercachepath
return os.path.join(_usercachedir(ui), hash)
File "/usr/lib/python2.7/dist-packages/hgext/largefiles/lfutil.py", line 97, in _usercachedir
raise error.Abort(_('unknown %s usercache location') % name)
Abort: unknown largefiles usercache location
Tried really hard whole day that and another tweak and advice but no luck :/
Some one with smart advice ?
PS: Also tried to clone old >> push new some repos and not good again

Partial resolve the problem - users cloned repositories can now make hg push repo
Add to /etc/mercurial/hgrc except:
[extensions]
largefiles =
and
[largefiles]
usercache = /path/to/largefiles/storage
Still facing the problem how to migrate all the repost from old to new server.
If I found proper solution will edit the topic.

Related

SOLVED: Remote mercurial server fails in changegroup.notify with an exception

Pushing changes to a remote mercurial repo is reporting an error in my changegroup.notify hook. It worked with an old 2.7 release of mercurial, but is broken now.
My server has
Python 3.9, hg --version is
Mercurial Distributed SCM (version 6.3.1)
The relevant repo's hgrc has
changegroup.notify = python:hgext.notify.hook
The hook reports an exception:
>: hg --traceback push
pushing to https://[redacted]
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: error: changegroup.notify hook raised an exception: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
remote: (run with --traceback for stack trace)
Why the exception, when I am using what is supposed to just work
out of the box?
Update: Re-ran the test pushing from an account on the same machine as the server (thanks #LazyBadger), got the following traceback:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/mercurial/hook.py", line 116, in pythonhook
r = obj(ui=ui, repo=repo, hooktype=htype, **pycompat.strkwargs(args))
File "/usr/local/lib/python3.9/dist-packages/hgext/notify.py", line 589, in hook
n = notifier(ui, repo, hooktype)
File "/usr/local/lib/python3.9/dist-packages/hgext/notify.py", line 326, in __init__
self.stripcount = int(self.ui.config(b'notify', b'strip'))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Solution:
Turns out this is the error you get if you haven't updated your hgrc in years. Used to be that adding the hook to your [.]hgrc was all that was required. Now you have to enable notify as well, so, e.g.
[extensions]
notify=
[hooks]
changegroup.notify = python:hgext.notify.hook
You also need some things to allow the email notication to work, for example if you have personal email support on the same server:
[reposubs]
* = [my email on the server]
[email]
from = Mercurial Notification <noreply#mydomain>
[smtp]
host = localhost

Mock fails to download disable plugin patterns while building 32bit RPMs: 403 on repository 'local'

I am trying to build a few RPMs for 32 bit Fedora using the SRPM. I tried running Mock with:-
mock -r fedora-32-i386 rebuild src.rpm
It gives me the following error:
No matches found for the following disable plugin patterns: local, spacewalk
local 83 B/s | 199 B 00:02
Errors during downloading metadata for repository 'local':
- Status code: 403 for https://kojipkgs.fedoraproject.org/repos/f32-build/latest/i386/repodata/repomd.xml (IP: 38.145.60.21)
Error: Failed to download metadata for repo 'local': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
ERROR: Command failed:
# /usr/bin/dnf --installroot /var/lib/mock/fedora-32-i686-bootstrap/root/ --releasever 32 --setopt=deltarpm=False --allowerasing --disableplugin=local --disableplugin=spacewalk install dnf dnf-plugins-core --setopt=tsflags=nocontexts
No matches found for the following disable plugin patterns: local, spacewalk
local 83 B/s | 199 B 00:02
Errors during downloading metadata for repository 'local':
- Status code: 403 for https://kojipkgs.fedoraproject.org/repos/f32-build/latest/i386/repodata/repomd.xml (IP: 38.145.60.21)
Error: Failed to download metadata for repo 'local': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
I have tried disabling the fastest mirror in the conf file in /etc/dnf, with no luck. Is mock using a different repo file than my current installation? How do I fix this?
Mock is using configs from the directory /etc/mock/ so for fedora-32-i386 it is /etc/mock/fedora-32-i386.cfg This config contains yum config which is used during the installation of the buildroot.
More about Mock's configs: https://rpm-software-management.github.io/mock/configuration

Saltstack smtp.send_msg returning with attribute error

I set up a beacon that tracks disk usage on all the Ubuntu Server 16.04 minions on Ubuntu Server 16.04. We wanted a reactor that emails us when disk space reaches a certain point. This question focuses on the master's ability to send an email. When I run the following command on the salt-master:
sudo salt-call smtp.send_msg 'email#email.com' 'Test message 3' subject='Test subject 3' profile='smtp-default'
I get the following result:
[ERROR ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'str' object has no attribute 'get'
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in <module>
salt_call()
File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 391, in salt_call
client.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 58, in run
caller.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 134, in run
ret = self.call()
File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 204, in call
ret['return'] = func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/salt/modules/smtp.py", line 95, in send_msg
server = creds.get('smtp.server')
AttributeError: 'str' object has no attribute 'get'
Traceback (most recent call last):
File "/usr/bin/salt-call", line 11, in <module>
salt_call()
File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 391, in salt_call
client.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/call.py", line 58, in run
caller.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 134, in run
ret = self.call()
File "/usr/lib/python2.7/dist-packages/salt/cli/caller.py", line 204, in call
ret['return'] = func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/salt/modules/smtp.py", line 95, in send_msg
server = creds.get('smtp.server')
AttributeError: 'str' object has no attribute 'get'
This is my salt --versions-report output of the master that returns the error:
Salt Version:
Salt: 2017.7.1
Dependency Versions:
cffi: 1.10.0
cherrypy: unknown
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.18
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-93-generic
system: Linux
version: Ubuntu 16.04 xenial
I have uninstalled and reinstalled the salt-master and salt-minion and I have updated Python and the salt-master. I also created another salt-master and it ran into the same problem. What is the fix or solution to this?
I have discovered that the error has nothing to do with the installed Python libraries or if my system is updated or not. I found that I did not have an SMTP server set up and that I need to create an SMTP mail server or have an accessible SMTP mail server in order to send emails as notifications of the reactor system. I discovered that the solution was to set up an SMTP mail server and then create a .conf file in the /etc/salt/minion.d directory that referenced an SMTP mail server like so:
smtp-default:
smtp.sender: mail#example.com
smtp.server: 0.0.0.0
smtp.tls: false
After creating a .conf file similar to the one above and the profile of the salt-call command referring to the function declaration of the state file, I was able to send emails to the desired address.

Building Custom Platform on ElasticBeanstalk Packer Error

I am trying to build a custom platform using the provided example (NodePlatform_Ubuntu) at
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/custom-platforms.html#custom-platforms-pda
I installed EB CLI (my OS is MacOSX El Captain):
➜ custom-platform ebp --version
EB CLI 3.10.1 (Python 2.7.1)
For some reason, EB CLI insists on using Python 2.7.1 instead of the already installed Python 3.x (python3 command works), but this shouldn't matter according to http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb-cli3-install.html it supports Python 2.7
While "ebp init" works perfectly. the "ebp create" fails.
➜ custom-platform ebp create
Creating application version archive "app-170427_145319".
Uploading custom-platform/app-170427_145319.zip to S3. This may take a while.
Upload Complete.
Note: An environment called 'eb-custom-platform-builder-packer' has been created in order to build your application. This environment will not automatically be terminated and it does have a cost associated with it. Once your platform creation has completed you can terminate this builder environment using the command 'eb terminate'.
INFO: createPlatform is starting.
INFO: Initiated platform version creation for 'custom-platform/1.0.2'.
INFO: Creating Packer builder environment 'eb-custom-platform-builder-packer'.
ERROR: Packer environment eb-custom-platform-builder-packer is not available, current status: terminated.
INFO: Failed to create platform version 'custom-platform/1.0.2'.
This is the error:
ERROR: Packer environment eb-custom-platform-builder-packer is not available, current status: terminated.
More logs (debug mode):
2017-04-27 14:19:55,698 (DEBUG) ebcli.lib.aws : Response: {u'Events': [{u'PlatformArn': 'arn:aws:elasticbeanstalk:us-west-2:107875334514:platform/custom-platform/1.0.1', u'Message': "Failed to create platform version 'custom-platform/1.0.1'.", u'Severity': 'INFO', u'EventDate': datetime.datetime(2017, 4, 27, 11, 19, 52, 84000, tzinfo=tzutc()), u'RequestId': 'f762956d-2b3a-11e7-8cd7-c96ae1e26915'}, {u'PlatformArn': 'arn:aws:elasticbeanstalk:us-west-2:107875334514:platform/custom-platform/1.0.1', u'Message': 'Packer environment eb-custom-platform-builder-packer is not available, current status: terminated.', u'Severity': 'ERROR', u'EventDate': datetime.datetime(2017, 4, 27, 11, 19, 51, 861000, tzinfo=tzutc()), u'RequestId': 'f762956d-2b3a-11e7-8cd7-c96ae1e26915'}], 'ResponseMetadata': {'date': 'Thu, 27 Apr 2017 11:19:52 GMT', 'RetryAttempts': 0, 'HTTPStatusCode': 200, 'RequestId': '6f7dcea9-2b3b-11e7-8cd7-c96ae1e26915'}}
ERROR: Packer environment eb-custom-platform-builder-packer is not available, current status: terminated.
INFO: Failed to create platform version 'custom-platform/1.0.1'.
Note that the app-170427_145319.zip is successfully uploaded to S3.
Any idea what's wrong?
It is was failing with instance profile(role) aws-elasticbeanstalk-custom-platform-ec2-role not getting necessary role inline policy attached to it.
Because of lot of modifications and changes, apparently due to an EB CLI bug, the instance profile got corrupted and was not taking InstanceProfile ARN when tried to create a profile with same name "aws-elasticbeanstalk-custom-platform-ec2-role".
So, we went ahead and created a new one named "custom-platform" and attached necessary inline policy permissions I have mentioned in the case.
Once that IAM issues are sorted, we went onto packer bundle,did "ebp init" and then ran "ebp create" but with -ip flag and used this instance profile custom-platform, as in
ebp create -ip custom-platform

Is there any working example of buildbot with Mercurial

buildbot version being used is:
$ buildbot --version
Buildbot version: 0.8.3p1
Twisted version: 10.1.0
Checkconfig, gives me errors:
$ buildbot checkconfig
/usr/lib/python2.6/dist-packages/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
import MimeWriter, tempfile, rfc822
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/scripts/runner.py", line 1071, in doCheckConfig
ConfigLoader(configFileName=configFileName)
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/scripts/checkconfig.py", line 46, in __init__
self.loadConfig(configFile, check_synchronously_only=True)
File "/usr/local/lib/python2.6/dist-packages/buildbot-0.8.3p1-py2.6.egg/buildbot/master.py", line 883, in loadConfig
% (b['name'], n))
ValueError: builder runtests uses undefined slave example-slave
$
Here is one example i looked at :
http://agiletesting.blogspot.com/2006/02/continuous-integration-with-buildbot.html
This pertains to:
Buildbot version: 0.8.8
Twisted version: 13.2.0
I had some serious issues to get it working with a simple hg repo, while the same project worked fine with git and the appropriate functions. So here it is.
There are three places deal with our repo in master.cfg: changesources, schedulers and builders, with only changesources and builders that use mercurial specific functions.
In changesources section:
from buildbot.changes.hgpoller import HgPoller
therepo=HgPoller(repourl="/home/user/test/my_project/",
branch='default',
pollInterval=30,
workdir='myrepo')
c['change_source'] = []
c['change_source'].append(therepo)
Here I use HgPoller, instead of PBChangeSource. The latter is more sophisticated but also requires more configuration steps (provide a port and yet another username and password).
repourl must point to the root of your hg repository. Any URL that could be used for "hg pull" or "hg clone" is acceptable. This example involves a local repository, but it could be on a server, then you would specify http something or else.
The default branch on mercurial is 'default'. pollInterval=30 says every 30 seconds, check for a new commit (this is from a toy example, in reality >30 would be more suitable).
Now the builder, which builds after a commit is detected and passed on by the scheduler(s):
from buildbot.process.factory import BuildFactory
from buildbot.steps.source.mercurial import Mercurial
factory = BuildFactory()
#watch out: this function is Mercurial, NOT Hg
checkout_default = Mercurial(repourl="/home/user/test/my_project/",
defaultBranch='default',
branchType='inrepo',
haltOnFailure = True)
factory.addStep(checkout_default)
# then you add some build instructions and don't forget to import the necessary...
What explains why my thing did not work is that I did not specify defaultBranch and branchType. Those keywords are not the same as with Git(), so beware. This is a little tricky as I did not find them in the user manual online, but it's there if you take a moment within the python interpreter:
import buildbot
help(buildbot.steps.source.mercurial)
Also, note that this is the function Mercurial imported from buildbot.steps.source.mercurial, which is not the same Mercurial function that you would import imported from buildbot.steps.source.Mercurial. The latter is deprecated (or that which you would use on an older version). My thanks to tomprince from the IRC buildbot channel on freenode for pointing that out.
The example you looked at is very old; c['bots'] was renamed to c['slaves'] a while ago, as well as many more changes.
I'd suggest taking a look at the Buildbot manual for configuration:
http://buildbot.net/buildbot/docs/current/Configuration.html#Configuration
And possibly also the installation section, to make sure you did what was required to set up the more recent versions of BuildBot, not just older versions:
http://buildbot.net/buildbot/docs/current/Installation.html#Installation
One example that was offered was the IcedTea buildbot, which builds from Mercurial repos. Configuration is browsable here:
http://icedtea.classpath.org/hg/buildbot/file
You're also welcome to drop by #buildbot on irc.freenode.net for help.