Using closed-source dependencies with Maven - mercurial

I have a closed-source project that I would like to build using Maven. It has a dependency on two java libraries which are not available in any public repository that I've been able to find (libGoogleAnalytics.jar and FlurryAgent.jar in this case, but the question applies to any closed-source dependency).
I would like anyone in my organization to be able to build the application using the exact same versions of the dependencies that I use to build the application. This includes my colleagues and our build-server.
How do I manage closed-source dependencies that maven doesn't know how to resolve?
Obviously, I could go to each person's machine and manually execute "mvn install:install-file" to get the binary into their maven repository, but manually managing dependencies like that defeats the purpose of a dependency manager.
As per maven's Internal Repositories documentation, I could set up a repository server somewhere and put the binaries there, which all the developers would then access. But that means I have a new server to maintain (or at least a new website on an existing server). It also means I have to worry about permissions to ensure that outside parties can't access the repository. It also means I have to worry about backups and availability now so that developers don't run into hiccoughs if the repository is unavailable.
All of these problems would go away for me if I could somehow use our existing scm (hg in this case, but it could be git or svn or whatever) to store the dependencies. Our source control repository is backed up already, it will basically always be available to developers doing builds, and its permissions have already been dealt with.
But I haven't been able to figure out how to manage maven dependencies using hg yet, if this is even possible.

It turns out that Manfred's answer didn't quite work for me. The app compiled, but it did not run on my Android device because the required google analytics classes were missing.
Following the links he supplied, I discovered this solution which is actually a little cleaner and worked properly.
In summary, I added the following dependencies to my pom.xml. The groupId, artifactId, and version were all made up by me using reasonable values:
<dependencies>
...
<dependency>
<groupId>com.google.android.apps.analytics</groupId>
<artifactId>libGoogleAnalytics</artifactId>
<version>1.1</version>
</dependency>
<dependency>
<groupId>com.flurry</groupId>
<artifactId>FlurryAgent</artifactId>
<version>1.24</version>
</dependency>
</dependencies>
I then added a repository definition for where I'm storing the third party dependencies in my project's source tree:
<repository>
<id>third.party.closed.source.repo</id>
<url>file://${basedir}/../maven_repo_3rd_party</url>
</repository>
I then moved the jar files to the following location:
./maven_repo_3rd_party/com/google/android/apps/analytics/libGoogleAnalytics/1.1/libGoogleAnalytics-1.1.jar
./maven_repo_3rd_party/com/flurry/FlurryAgent/1.24/FlurryAgent-1.24.jar
Once I did that, my project compiled and ran exactly as if the third party dependencies were resolved from an official maven repo.

While I really think you should use a dedicated repository server and Sean Patrick is totally right about it here is a hack to get it to work.
Put the jar file in a libs folder just like you did in the days gone by (remember Ant.. ouch) .. and then declare a dependency to each jar using the scope system and a path.
An example can I did this for is described here
http://www.simpligility.com/2010/01/how-to-mavenize-a-typical-web-application-build-jasperserver-3-7-sample-webapp/
Specifically a dependency would e.g. look like this
<dependency>
<groupId>jasperreports</groupId>
<artifactId>jasperreports-chart-themes</artifactId>
<version>3.7.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/src/main/webapp/WEB-INF/lib/jasperreports-chart-themes-3.7.0.jar</systemPath>
</dependency
Oh and now that I told you how to do it keep in mind that this is BAD practice and has a bunch of issues but it will work...

Use A dedicated Repository Server
As per maven's Internal Repositories
documentation, I could set up a
repository server somewhere and put
the binaries there, which all the
developers would then access.
Exactly. Set up a maven repository server with several repositories, e.g. these:
internal-releases
internal-snapshots
external-opensource
external-closedsource (this is where the lib we are talking about goes)
But that means I have a new server to
maintain (or at least a new website on
an existing server). It also means I have
to worry about permissions to ensure that
outside parties can't access the repository.
Yes, but a company that does serious software development should have an infrastructure like that. But if your company is serious about using Maven, there should probably also be a dedicated position for configuration management, and that person should administer this server.
It also means I have to worry about
backups and availability now so that
developers don't run into hiccoughs if
the repository is unavailable.
The standard repository servers (e.g. Sonatype Nexus) are rock solid. If it ever hangs, just restart the app server / servlet container it's running on. Also, once developers have downloaded a library from the repo, it remains in the local repo, so even if the repo is down, there shouldn't be a problem (but you can't reference a new dependency when the server is down).
Use your existing SCM as a maven repository
OK, if you really want to use your SCM as a maven repo, here's how to do it:
http://maven-svn-wagon.googlecode.com/svn/site/index.html
This article describes how to setup an SVN-based maven repository for your own project. But if you want to deploy a third-party to the repo, just create a pom with the config mentioned here and use that pom to deploy:deploy-file your library.
(There are other wagon / scm implementations also, and the configuration is slightly different, but the solution remains the same: create a pom according to the wagon implementation you are using and then do deploy:deploy-file (see more info on the usage page)

Related

CAS overlay: How to reduce the WAR size

I am upgrading CAS from 3.x to 5.x. The CAS Overlay Installation: builds a war file of about 120 MB about 5 times larger than 3.x.
How can I skip the unused modules/jars from the war file in order to reduce the file size and deployment time? I only need to authenticate against a Database and AD/LDAP.
The build for CAS 5.3.x is based on Apache Maven, and Apache Maven has a concept for excluding dependencies. You can read more about this here. This means that you'd have to find the coordinates for dependencies you think you don't need and exclude from your deployment via exclusions tag. Furthermore, the Apache WAR plugin does allow you exclude files from the final packaging, which you might also be able to use to remove JARs, etc from the final web application artifact that you think you don't need. You can read more about that here.
Tweaking the internals and list of dependencies is a completely non-supported approach and may directly affect the stability and security of the deployment. Do so at your own risk.

DIY Tomcat, where is the webapp folder on the local git repo

following OpenShift tutorials, creating a tomcat application and clone it, the local repository will contain a pom.xml, webapp folder.
What's the equivalent on a diy application that contain a diy and misc folders
Thank's in advance, any help is appreciated cause I'm really stuck here !?
Update
Well I've install a Tomcat 8 DIY application following this tutorial here everything works fine, I can see the Tomcat page in the browser, the problem is how to deploy a .war file.
For a Tomcat 6/7 application on OpenShift, the local git repository have this structure:
____Tomcat7/6
|_________ webapp
|_________ src
|_________ pom.xml
But for a Tomcat 8 DYI application I have this structure
________Tomcat8/diy
|__________ Diy
|__________ misc
|__________ readme
So Where to deploy my .war files, cause there is no webapp folder?
The title of your question suggests that you are mixing up at least 5 different, completely independent & orthogonal tools and concepts:
Git is a version control system ("push", "local repo")
Maven is a build tool ("pom.xml")
Apache Tomcat is a servlet container ("Tomcat 6/7/8")
rhc is some client tool provided by yet another cloud computing platform ("OpenShift")
Your code is the stuff that you have written, it's completely under your responsibility.
Before you start doing anything, please make sure that you have at least some basic understanding of what each of these tools does. Then ask yourself whether you really need Tomcat 8 instead of Tomcat 7, and whether a 2 year blog post about the compilation of Tomcat 8 within an OpenShift gear is the best source. All these deployment details can change pretty quickly, if it worked two years ago, it's not guaranteed that it would work now.
I've never worked with OpenShift, but as far as I understand, the basic idea is this:
You write your code
You create your OpenShift account and allocate some "Gear" (or "Dyno" or whatever...) for your application
You commit your source code (/src) and the files that are necessary for the build (pom.xml), and use git to push it to the repository OpenShift gave you.
OpenShift then uses your pom.xml and builds all the WAR-files on it's own
Then you can use your rhc client tool to start your application, if that's not done automatically.
Some of these steps can be changed.
If you really have to, you can indeed compile your own Tomcat8, the tutorial you linked tells you how (more or less. The dude who did it obviously knew what he was doing there, so he might have skipped some details that seemed trivial to him).
Furthermore, if you really want, you can deploy pre-packaged WAR-files, by deliberately removing all the stuff that is necessary to build you app (removing pom.xml and all the /src), and instead adding the packaged application to your git repo, and then pushing it all to OpenShift. Then it will skip the build step, and just run what you gave it. OpenShift seems to provide some information about this deployment strategy: https://help.openshift.com/hc/en-us/articles/202399740. Please read the documentation and make sure you understand what you want to do. For example, filter-branching your git repo and removing all source files you have ever written is not a good idea, even if you don't need these files on OpenShift.
Currently, I don't see anything of the standard tomcat directory structure in the tree that you show. Instead, there seem to be just some basic ruby-scripts or some other default-demo-app-stuff... That's why it's called "do it yourself". If you don't want this, take a standard Tomcat7 app.

m2eclipse on windows with Mercurial does not allow hg to be used on the "Check out Maven Projects from SCM" wizard

I followed the answer from this posting:
How to use Mercurial, Maven and Eclipse together?
But the "Check out Maven Projects from SCM" still does not allow me to use Mercurial SCM.
The only option I get is "svn" in the dropdown, and even if I ignore the drop down and enter in "scm:hg:http://myMercurialRepoURL"
I'm using:
Eclipse 3.6.1
m2eclipse 0.12 from http://m2eclipse.sonatype.org/sites/m2e
"Maven SCM handler for Subclipse" 0.12.0 from m2eclipse extra's site (sorry, I would give the URL, but my new account doesn't have enough rep)
MercurialEclipse 1.8.1 from http://cbes.javaforge.com/update
And I've uninstalled, and reinstall those plugins in that order.
I'm able to use Maven in projects, and use Mercurial separately in Eclipse - it's just this one Wizard that seems to be broken, which leads me to believe that if I try to Materialize an Artifact from a Mercurial repo that it will also fail.
Has anyone had any luck with this Wizard? Perhaps on earlier versions? Is this a new bug?
Thanks
As far my search went, I didn't get to find a suitable connector for mercurial and m2eclipse (in terms of only using IDE). However, I did an experiment wherein I cloned a copy of the source from outside the IDE. (via TortoiseHg specifically)
Afterwards, assuming you have already m2e installed in Eclipse:
Go to File > import > maven > existing maven projects, then select the folder where you've originally pulled/cloned your source code.
I believe by doing so, you'll see the m2e commands in the project's context menu (via run as), and effective use mercurial commands. (via team context menu)
At least for now, this is better not using m2e and mercurial at the same time.
If anyone has a more streamlined approach, I'm also curious. :D

Mercurial repository usage with binary files for building setup files

I have an existing Mercurial repository for a C++ application in a small corporate environment. I asked a co-worker to add the setup script to the repository and he added all of the dependency binaries, PDFs, and executable to the repository under an Install directory. I dislike having the binaries and dependencies in the same repository, but I'd like recommendations on best practices. Here are the options I am considering:
Create a separate repository for the
Installer and related files
Create a subrepository for the
Installer and related files
Use a (yet to be identified) build
dependency manager
I am concerned with using a subrepository with Mercurial based on what I've read so far and the (apparently) incomplete implementation. I would like to get a project dependency system, e.g. Ivy, but I don't know all of the options and haven't had time yet to try out any options.
I thought I'd use TortoiseHg as a basis, and it does not have the TortoiseHg binaries in the repository although it does have some binaries such as kdiff3.exe. Instead it uses setup.py to clone multiple repositories and build the apps. This seems reasonable for OSS, but not so much for corporate environments.
Recommendations?
I've had great luck using a good dependency manager, but it's more useful for code modules than artifacts. A sub-repo certainly works, but you'll not be cutting the clone time of people pulling the top tree since it will cascade into the subrepo too.
Someone will probably suggest using bfiles or big files, but I'd avoid it. They work well enough, but that functionality looks like it's going into mercurial this summer natively at which time those will languish.
Were I you I'd (a) hit the co-worker (b) try to strip those added files (using strip or convert), (c) re-do it will the installers being built not stored.
The PDFs I'd probably store, but there are some neat tricks for generating docs if you're interested.

How can one configure Hudson to integrate with Sventon?

Hudson ver. 1.353
Sventon ver. 2.14
I just cannot figure out how to configure Hudson to work with Sventon. It seems that the path format that Hudson expects from Sventon is not the format used by Sventon.
Any ideas?
Thanks.
UPDATE
Given an SVN repository with the name of windows, the Sventon URL path to the repository is http://dev-builder:8080/svn/repos/windows/list/
However, Hudson expects something like
http://dev-builder:8080/svn/repobrowser.svn?name=windows
Can anyone explain how this should be configured?
Regarding configuration, under the Source Code Management section of a job configuration, the Repository Browser dropdown lists Sventon 2.x as one of the options. (Not trying to be snarky, just making sure you're using the correct configuration.)
There are some Hudson bugs (search for sventon) in various states that might be related to your issue.
It's not clear to me whether this is a configuration problem or a Hudson bug. You could post the relevant configuration and both the paths that Hudson generates and Sventon expects. If it is a reproducible Hudson bug, reporting it to the Hudson bug database is the best bet.
Update with my experience: Under Source Code Management, I configured my Repository Browser to be Sventon 2.x and set the Repository URL to http://localhost:8080/svn and Repository Instance to windows. Hudson then listed changes with Sventon links as http://localhost:8080/svn/repos/windows/info?revision=XYZ
I think this means you should set:
Repository Browser to Sventon 2.x
Repository URL to http://dev-builder:8080/svn, and
Repository Instance to windows
Beware that the Hudson inline documentation for Sventon 2.x is wrong about the URLs that will be generated. It looks like this was never updated from 1.x.
You should however be aware that Sventon does not integrate well with projects that have multiple modules from different repositories. We for example use repositories A, B and C. And we've set our repository instance to A so that we can browse to things in that repository but not in any of the other.
Otherwise I really like Sventon as a svn browser.