tom callaway (spot) wrote,
tom callaway
spot

How you know your Free or Open Source Software Project is doomed to FAIL (or at least, held back fro

This was inspired by my recent efforts to look at Chromium, but these are just some of the red flags I generally have observed over the years written down.

== Size ==
* The source code is more than 100 MB. [ +5 points of FAIL ]
* If the source code also exceeds 100 MB when it is compressed [ +5 points of FAIL ]

== Source Control ==
* There is no publicly available source control (e.g. cvs, svn, bzr, git) [ +10 points of FAIL ]
* There is publicly available source control, but:
* There is no web viewer for it [ +5 points of FAIL ]
* There is no documentation on how to use it for new users [ +5 points of FAIL ]
* You've written your own source control for this code [ +30 points of FAIL ]
* You don't actually use the existing source control [ +50 points of FAIL ]

== Building From Source ==
* There is no documentation on how to build from source [ +20 points of FAIL ]
* If documentation exists on how to build from source, but it doesn't work [ +10 points of FAIL ]
* Your source is configured with a handwritten shell script [ +10 points of FAIL ]
* Your source is configured editing flat text config files [ +20 points of FAIL]
* Your source is configured by editing code header files manually [ +30 points of FAIL ]
* Your source isn't configurable [ +50 points of FAIL ]
* Your source builds using something that isn't GNU Make [ +10 points of FAIL ]
* Your source only builds with third-party proprietary build tools [ +50 points of FAIL ]
* You've written your own build tool for this code [ +100 points of FAIL ]

== Bundling ==
* Your source only comes with other code projects that it depends on [ +20 points of FAIL ]
* If your source code cannot be built without first building the bundled code bits [ +10 points of FAIL ]
* If you have modified those other bundled code bits [ +40 points of FAIL ]

== Libraries ==
* Your code only builds static libraries [ +20 points of FAIL ]
* Your code can build shared libraries, but only unversioned ones [ +20 points of FAIL ]
* Your source does not try to use system libraries if present [ +20 points of FAIL ]

== System Install ==
* Your code tries to install into /opt or /usr/local [ +10 points of FAIL ]
* Your code has no "make install" [ +20 points of FAIL ]
* Your code doesn't work outside of the source directory [ +30 points of FAIL ]

== Code Oddities ==
* Your code uses Windows line breaks ("DOS format" files) [ +5 points of FAIL ]
* Your code depends on specific compiler feature functionality [ +20 points of FAIL ]
* Your code depends on specific compiler bugs [ +50 points of FAIL ]
* Your code depends on Microsoft Visual Anything [ +100 points of FAIL ]

== Communication ==
* Your project does not announce releases on a mailing list [ +5 points of FAIL ]
* Your project does not have a mailing list [ +10 points of FAIL ]
* Your project does not have a bug tracker [ +20 points of FAIL ]
* Your project does not have a website [ +50 points of FAIL]
* Your project is sourceforge vaporware [ +100 points of FAIL ]

== Releases ==
* Your project does not do sanely versioned releases (Major, Minor) [ +10 points of FAIL ]
* Your project does not do versioned releases [ +20 points of FAIL ]
* Your project does not do releases [ +50 points of FAIL ]
* Your project only does releases as attachments in web forum posts [ +100 points of FAIL ]
* Your releases are only in .zip format [ +5 points of FAIL ]
* Your releases are only in OSX .zip format [ +10 points of FAIL ]
* Your releases are only in .rar format [ +20 points of FAIL ]
* Your releases are only in .arj format [ +50 points of FAIL ]
* Your releases are only in an encapsulation format that you invented. [ +100 points of FAIL ]
* Your release does not unpack into a versioned top-level directory (e.g. glibc-2.4.2/ ) [ +10 points of FAIL ]
* Your release does not unpack into a top-level directory (e.g. glibc/ ) [ +25 points of FAIL ]
* Your release unpacks into an absurd number of directories (e.g. home/johndoe/glibc-svn/tarball/glibc/src/) [ +50 points of FAIL ]

== History ==
* Your code is a fork of another project [ +10 points of FAIL ]
* Your primary developers were not involved with the parent project [ +50 points of FAIL ]
* Until open sourcing it, your code was proprietary for:
* 1-2 years [ +10 points of FAIL ]
* 3-5 years [ +20 points of FAIL ]
* 6-10 years [ +30 points of FAIL ]
* 10+ years [ +50 points of FAIL ]

== Licensing ==
* Your code does not have per-file licensing [ +10 points of FAIL ]
* Your code contains inherent license incompatibilities [ +20 points of FAIL ]
* Your code does not have any notice of licensing intent [ +30 points of FAIL ]
* Your code doesn't include a copy of the license text [ +50 points of FAIL ]
* Your code doesn't have a license [ +100 points of FAIL ]

== Documentation ==
* Your code doesn't have a changelog [+10 points of FAIL]
* Your code doesn't have any documentation [ +20 points of FAIL ]
* Your website doesn't have any documentation [ +30 points of FAIL ]

=== FAIL METER ===
0 points of FAIL: Perfect! All signs point to success!
5-25 points of FAIL: You're probably doing okay, but you could be better.
30-60 points of FAIL: Babies cry when your code is downloaded
65-90 points of FAIL: Kittens die when your code is downloaded
95-130 points of FAIL: HONK HONK. THE FAILBOAT HAS ARRIVED!
135+ points of FAIL: So much fail, your code should have its own reality TV show.

Anyone want to guess how many POF (points of FAIL) Chromium has?
  • Post a new comment

    Error

    Anonymous comments are disabled in this journal

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

  • 50 comments
It would be interesting to see what the score of most packages would be using that criteria (lets just say that we change the points to +1 per item.)
ha!
Finally an objective criteria to actually organize stuff in the review que.
When searching for stuff to review for inclusion I'd love to list them by FAIL number so I can concentrate my limited time on helping the lower FAIL projects into Fedora.

-jef
I recently started my first own project, and I totally find some of my mistakes in this list (let me be naive and assume that those mistakes come from my lack of experience in project management).

Thanks for that, I'll sure use it to correct them and try to prevent more kittens from dying :D
* The source code is more than 100 MB. [ +5 points of FAIL ]
* If the source code also exceeds 100 MB when it is compressed [ +5 points of FAIL ]
I think these are a little harsh on the large successful projects.
- qt-x11-opensource-src-4.5.1.tar.bz2 is 110MiB.
- Lots of the KDE tarballs exceed 100MiB of source code.
- gcc and the kernel these days exceeds 100MiB of source.

* Your code depends on specific compiler feature functionality [ +20 points of FAIL ]
- There's lots of GCC-specific code out there in terms of other compilers being broken. Even the kernel has it.

Your source building options seem hyper-specific to C/C++. Ant for Java projects, setuptools for Python, MakeMaker for Perl are acceptable.

I'd add fail points:
- for building static libraries with -fPIC objects.
- Not using the changelog in the release (worse than not having it I'd say)
- Having the autoconf/automake template NEWS/README files.
It's only five points of fail. Certainly the scale should be calibrated.

robbat2

7 years ago

PIC static libs

kevin_kofler

7 years ago

Re: PIC static libs

robbat2

7 years ago

Re: PIC static libs

robbat2

7 years ago

Re: PIC static libs

robbat2

7 years ago

Re: PIC static libs

robbat2

7 years ago

I'm guessing you're in the dead kittens range ???
I've definitely worked on projects in that range.
Oh, another whole set involves language.
The code is in English but the docs are in Spanish - 5 pts of Fail.
The code is in English but has comments in Spanish - +5 pts of Fail.
The code uses Spanish variable names, but the Docs are only available in English OMG WHAT ARE YOU SMOKING ?
English, Spanish, a couple of other languages of western and central European descent, I can handle them fine.

But, when it hits Cyrillic written in the Volapuk encoding, then we're into fail range. Or worse, Tamil or various Indonesian scripts.

reddragdiva

5 years ago

Congrats ...

Anonymous

May 30 2009, 04:09:49 UTC 7 years ago

... you just killed most Java code bases. I'm serious. Congratulations!
For all I can see, this applies roughly to the linux kernel:

    * The source code is more than 100 MB.                          [   +5 points of FAIL ]
    * If the source code also exceeds 100 MB when it is compressed  [   +5 points of FAIL ]
    * You've written your own source control for this code          [  +30 points of FAIL ]
    * Your source is configured editing flat text config files      [  +20 points of FAIL ]
    * You've written your own build tool for this code              [ +100 points of FAIL ]
    * Your project does not have a bug tracker                      [  +20 points of FAIL ]
    * Your code doesn't have a changelog                            [  +10 points of FAIL ]
    * Your code doesn't have any documentation                      [  +20 points of FAIL ]


Which makes 210 as a total ;)
I don't know if git counts. It may have been written with Linux as it's primary use case but it's used for plenty of other projects and indeed is spun off and separate from the kernel code base.

The kernel also uses Make although I'll accept the menuconfig/config stuff does add a little complexity.

There is a fair amount of Documentation in Documentation/
You need a score for "Your code has a licence you made up".
I think thats at least 100 points of fail.
Please let us know how Chromium scores... :-)
* The source code is more than 100 MB. [ +5 points of FAIL ]
- Chromium source tarball decompressed to 3Gbytes here (according to a simple du). +5
* If the source code also exceeds 100 MB when it is compressed [ +5 points of FAIL ]
- Chromium source tarball looks to be 645Mbytes when compressed. +5

== Source Control ==
* There is no publicly available source control (e.g. cvs, svn, bzr, git) [ +10 points of FAIL ]
- There is a public svn repo: http://src.chromium.org/svn/trunk/src/ (how safe it is to check it out I don't know though - seems to work with svn co but this isn't the advised route: http://dev.chromium.org/developers/how-tos/get-the-code )
* There is publicly available source control, but:
* There is no web viewer for it [ +5 points of FAIL ]
- You can use an ordinary web browser to see basic directories in SVN: http://src.chromium.org/svn/trunk/src/ .
* There is no documentation on how to use it for new users [ +5 points of FAIL ]
- The gclient depot tool is under documented: http://dev.chromium.org/developers/how-tos/install-gclient .
- There appears to be documentation
* You've written your own source control for this code [ +30 points of FAIL ]
- Yeah, something like this seems to be happening with gclient... +30
* You don't actually use the existing source control [ +50 points of FAIL ]
- Yeah, something like this seems to be happening with gclient... +50

== Building From Source ==
* There is no documentation on how to build from source [ +20 points of FAIL ]
- There are some build instructions on http://code.google.com/p/chromium/wiki/LinuxBuildInstructions .
* If documentation exists on how to build from source, but it doesn't work [ +10 points of FAIL ]
- I haven't tried building it but from the comments on that page this would appear to be the case +10.
* Your source is configured with a handwritten shell script [ +10 points of FAIL ]
- Doesn't sound like it.
* Your source is configured editing flat text config files [ +20 points of FAIL]
- Don't know.
* Your source is configured by editing code header files manually [ +30 points of FAIL ]
- Don't know.
* Your source isn't configurable [ +50 points of FAIL ]
- Don't know.
* Your source builds using something that isn't GNU Make [ +10 points of FAIL ]
- I have a feeling chromium uses SCons. +10.
* Your source only builds with third-party proprietary build tools [ +50 points of FAIL ]
- I don't think this is the case.
* You've written your own build tool for this code [ +100 points of FAIL ]
- Doesn't sound like it.

== Bundling ==
* Your source only comes with other code projects that it depends on [ +20 points of FAIL ]
- Not quite sure what this means.
* If your source code cannot be built without first building the bundled code bits [ +10 points of FAIL ]
- I think this is the case +10
* If you have modified those other bundled code bits [ +40 points of FAIL ]
- I think this might be the case +40

== Libraries ==
* Your code only builds static libraries [ +20 points of FAIL ]
- Don't know.
* Your code can build shared libraries, but only unversioned ones [ +20 points of FAIL ]
- Don't know.
* Your source does not try to use system libraries if present [ +20 points of FAIL ]
- Don't know.

== System Install ==
* Your code tries to install into /opt or /usr/local [ +10 points of FAIL ]
- Don't know.
* Your code has no "make install" [ +20 points of FAIL ]
- Since it uses SCons this is unlikely. +20
* Your code doesn't work outside of the source directory [ +30 points of FAIL ]
- Don't know.

== Code Oddities ==
* Your code uses Windows line breaks ("DOS format" files) [ +5 points of FAIL ]
- On the files I peeked at vi didn't say the line endings were DOS.
* Your code depends on specific compiler feature functionality [ +20 points of FAIL ]
- Don't know (I'm running a build of Chromium so one guesses its portable to at least MSVC and gcc).
* Your code depends on specific compiler bugs [ +50 points of FAIL ]
- Don't know.
* Your code depends on Microsoft Visual Anything [ +100 points of FAIL ]
- Well if you have Windows only parts how do you avoid this?
== Communication ==
* Your project does not announce releases on a mailing list [ +5 points of FAIL ]
- There is an announcement mailing list http://groups.google.com/group/chromium-announce (although it's currently quiet).
* Your project does not have a mailing list [ +10 points of FAIL ]
- There are mailing lists via google groups - http://dev.chromium.org/developers/discussion-groups .
* Your project does not have a bug tracker [ +20 points of FAIL ]
- There is a bug tracker - http://code.google.com/p/chromium/issues/list .
* Your project does not have a website [ +50 points of FAIL]
There is a website - http://dev.chromium.org/Home .
* Your project is sourceforge vaporware [ +100 points of FAIL ]
- Well I am running a build and it's not hosted on sourceforge : )
== Releases ==
* Your project does not do sanely versioned releases (Major, Minor) [ +10 points of FAIL ]
- I don't think this has happened so far (but it's still in alpha so...)
* Your project does not do versioned releases [ +20 points of FAIL ]
- I don't think this has happened so far (but it's still in alpha so...)
* Your project does not do releases [ +50 points of FAIL ]
- I don't think this has happened so far (but it's still in alpha so...)
* Your project only does releases as attachments in web forum posts [ +100 points of FAIL ]
- I don't think this has happened so far (but it's still in alpha so...)
* Your releases are only in .zip format [ +5 points of FAIL ]
- I don't think this has happened so far (but it's still in alpha so...)
* Your releases are only in OSX .zip format [ +10 points of FAIL ]
- What's OSX zip format?
* Your releases are only in .rar format [ +20 points of FAIL ]
- I don't think this has happened so far (but it's still in alpha so...). What about corrupt multipart rars? : )
* Your releases are only in .arj format [ +50 points of FAIL ]
- I doubt this is likely.
* Your releases are only in an encapsulation format that you invented. [ +100 points of FAIL ]
- I doubt this is likely.
* Your release does not unpack into a versioned top-level directory (e.g. glibc-2.4.2/ ) [ +10 points of FAIL ]
- tarball is effectively a "nightly svn snapshot" so + 10
* Your release does not unpack into a top-level directory (e.g. glibc/ ) [ +25 points of FAIL ]
- Dunno quite what that means...
* Your release unpacks into an absurd number of directories (e.g. home/johndoe/glibc-svn/tarball/glibc/src/) [ +50 points of FAIL ]
- Yup: home/chrome-svn/tarball/chromium/src/ +50

== History ==
* Your code is a fork of another project [ +10 points of FAIL ]
- Kinda. It includes parts of webkit.
* Your primary developers were not involved with the parent project [ +50 points of FAIL ]
- Dunno.
* Until open sourcing it, your code was proprietary for:
* 1-2 years [ +10 points of FAIL ]
* 3-5 years [ +20 points of FAIL ]
* 6-10 years [ +30 points of FAIL ]
* 10+ years [ +50 points of FAIL ]
- Dunno.

== Licensing ==
* Your code does not have per-file licensing [ +10 points of FAIL ]
- Don't know what this means - every file has to have a copyright notice?
* Your code contains inherent license incompatibilities [ +20 points of FAIL ]
- Dunno.
* Your code does not have any notice of licensing intent [ +30 points of FAIL ]
There is a LICENSE file.
* Your code doesn't include a copy of the license text [ +50 points of FAIL ]
There is a LICENSE file.
* Your code doesn't have a license [ +100 points of FAIL ]
There is a LICENSE file.

== Documentation ==
* Your code doesn't have a changelog [+10 points of FAIL]
Does svn count?
* Your code doesn't have any documentation [ +20 points of FAIL ]
- Dunno.
* Your website doesn't have any documentation [ +30 points of FAIL ]
There appears to be documentation on http://dev.chromium.org/developers .

How about "your binary releases only target x86 CPUs with modern features"?

Re: POF for Chromium (cont)

Anonymous

7 years ago

Re: POF for Chromium (cont)

Anonymous

7 years ago

Re: POF for Chromium

Anonymous

7 years ago

So... you didn't like it much, eh? ;-)

That's a useful checklist -- worth putting up somewhere more permanent! Here are a few more, mostly from an article I wrote about packaging Unix software a couple of years ago. I've seen all of these in the wild -- many from otherwise-reputable projects...

  • Your distribution files don't have predictable names or download URLs. (+10)
  • Your distribution files can't be automatically downloaded. (+20)
  • Your distribution tarball isn't signed. (+10)
  • Your distribution tarball is signed, but the key is only available from your website. (+20)
  • Your distribution tarball is signed, but the key isn't available from anywhere. (+50)
  • Your distribution is signed, but in a weird way (e.g. you signed the output of md5sum). (+10)
  • Your releases are only in the format of 2GB DVD image files. (+50)
  • Your code only builds statically-linked, stripped binaries. (+10)
  • Your code only builds statically-linked, stripped binaries, and tends to crash a lot. (+30)
  • The files in your distribution have ridiculous permissions (e.g. world-writable files, or non-executable directories). (+10)
  • Your distribution contains editor backup files, version control system internal state, etc. (+10)
  • Your distribution contains object files and/or executables that you forgot to clean out. (+30)
  • Your distribution has only been tested on a case-insensitive filesystem. (+50)
  • Your source cannot be configured automatically (e.g. there's only a curses- or GUI-based configuration frontend). (+50)
  • Your build system doesn't allow compiler flags and/or environment variables to be overridden. (+30)
  • Your build system depends on csh. (+50)
  • Your build system fails to detect when something's gone wrong during building or installation. (+10)
  • Your build system has to do the entire build as root; installation isn't a separate step. (+20)
  • Your build system only supports installation as root. (+20)
  • Your build system doesn't support DESTDIR-staged installation. (+10)
  • Your build system mostly supports DESTDIR, except for one installation rule where you forgot. (+50)
  • Your build system doesn't support changing the installation prefix at all. (+30)
  • Your build system uses variable names like "prefix" and "DESTDIR" in non-standard ways. (+20)
  • Your build system doesn't make sure directories exist before installing things into them. (+10)
  • Your build system overwrites existing configuration files without asking. (+20)
  • Your build system edits configuration files installed by other packages. (+30)
  • Your build system replaces executables installed by other packages. (+50)
  • Your code tries to work out its installation prefix at runtime. (+10)
  • Your code tries to work out its installation prefix at runtime, and fails miserably in the presence of symbolic links. (+30)
  • Your code doesn't have a test suite. (+10)
  • Your code has a test suite, but it doesn't pass it as distributed. (+50)
> # Your distribution is signed, but in a weird way (e.g. you signed the output of md5sum). (+10)
Just out of interest, what's your objection to this?

The GPG forces you to a single digest, doesn't protect the original filename, and forces you to use GPG even if you just want to check that the download was intact, not the authenticity of it, thus you can use sha1sum/md5sum -c still, or any other program and check the values yourself.

Here's a combined SHA1/MD5 signed digest.
http://distfiles.gentoo.org/releases/amd64/2008.0/livecd/livecd-amd64-installer-2008.0-r1.iso.DIGESTS

Signing the .DIGESTS is also a hell of a lot faster than waiting for GPG to suck in the entire iso/tarball.

Internally the default digest that GPG uses is still SHA1, so all I can see is the potential for maybe one more place to conduct an attack against the digest - 1 if only .asc, maybe 2 if .DIGEST. I say maybe 2 as the .DIGEST base layer is as strong as the strongest hash only. As soon as there the SHA-NG competition is completed, we can add that to our .DIGEST files without breaking backwards compatibility.

Re: A few more

atsampson

7 years ago

Hey, you added rsync to mirror.ac.uk!

Anonymous

7 years ago

Re: A few more

spot

7 years ago

Re: A few more

iquaid

7 years ago

Are we counting the Chromium test cases against it? I think a lot of the code is test cases; at which point you're either penalizing good software practices or planning to avoid them yourself (shipping a build without running test cases)
I agree with the poster that says Ant for Java, etc.
I started trying to apply these points to an Ajax project. Most successful Ajax projects are distributed as .zip files for example. I can't imagine they should get fail points for that. Would anybody even want a jQuery RPM? Obviously autoconf, GNU Make, library comments and 'make install' don't apply either.

For iPhone projects such as Three20, they should probably get FAIL points if they don't use XCode. The general rule is you should use the standard tools for your target market/developer. Prefer Open Source when all else is equal, but if it's for the iPhone you should probably use XCode.

Then again, maybe your definition of FAIL is that it won't get packaged as an RPM and included with Linux.

There's a lot of wisdom in these guidelines and I'd be interested in reviewing any version that is made more generic. Thanks for posting them!
Overall, I think there is a major component that is, "Don't forget that you are going to need to be running on an OS and in a way that is sane for people who might be building what matters to them around you."

So, yes, because Linux is the dominant web hosting platform, being packaged in a sane way (RPM, DEB) is a reasonable goal.

You want your system administrator to be able to duplicate, mirror, and migrate live instances of your application in to magic cloud space. If you make your sysadmin manage another package format (ZIP) where it bundles other JARs/JavaScript, the sysadmin now has an entirely new vector to watch for security and bug fixes. All the rest of the system has everything running through a common package format, using common system libraries, etc., except for your one, cool web application. That requires a special process and watching all of its own. Hopefully the bundled binary bits are all properly open source and redistributable by you.

I get that being able to unzip a jarball, reload Tomcat, and start developing is attractive to developers. But there is eventually going to be a moment beyond all that where you want people to care about what you are doing. Yes, we've shown that people, especially developers, will come check it out if it is ultra-cool even if it involves a custom build/package/deploy method. But you've handed off a tough uphill battle in the name of agility and ease. It is trading future flexibility and security for apparent speed in the present. It is pushing the complexity of sustainability to the developer who was hoping to just concentrate on their own app.

If you want the 17 million+ users of Fedora Linux (for example) to be able to go to System > Administration > Add/Remove Software and find your software there, to be able to use it, test it, help contribute to improving it, then you are going to want it in a package format that Fedora can consume. Go the extra mile and package it for Debian, and you have probably doubled the Linux install base potential.

For something like JavaScript, I reckon the packaging standards are still evolving. Old rules cannot always cover new situations. But the solution isn't to invent an entirely new game from scratch, especially without trying to work with a major Linux distro on a workable solution.

Anonymous

April 14 2010, 15:21:02 UTC 7 years ago

* There is no publicly available source control (e.g. cvs, svn, bzr, git) [ +10 points of FAIL ]
* Your project does not have a bug tracker [ +20 points of FAIL ]
See TangoGPS and the recent performance of 'The community can't be trusted with official source control/bug tracker' - http://permalink.gmane.org/gmane.comp.handhelds.openmoko.community/55915

You also don't have "Your project does not accept patches" 5 points or "Your project accepts patches but has an explicit policy of only merging those from the cabal of 'core' developers (http://pidgin.im/cgi-bin/mailman/listinfo/cabal)" +50... Yes, I'm looking at you Pidgin.
I liked this list however I'm not too sold on
Your code tries to install into /opt or /usr/local [ +10 points of FAIL ]

Google Chrome, Sublime, VMWare and various games install to /opt - Seems silly to me.
Nowhere in the list does QUALITY come into play aside from having a bug tracking system. Needs to add "No publicly transparent process in place for bug fixing [ +100 points of FAIL ]" and also "New features always take precedence over bug fixing [ +50 points of FAIL ]". I guess for both open and closed source software projects quality is still considered optional and the first thing to get crossed off the list. Sadly, users put up with that.

I also wonder how the size limit is measured. Does that exclude code comments? I like code that has a lot of comments explaining in plain simple language what the next 3 to 5 lines of code do. It is much easier to decipher than trying to compile a programming language intended for machines in my head.

Lastly, that list arrogantly assumes that there is no open source software created on Windows and for Windows. Even using closed source runtimes that are freely available and freely distributable are not a problem. In the end one big metric for success is always ROI. That applies to open or closed source applications. If it is a pain to install and use forget about it. So let me propose one more thing for the list:
Not providing a commonly used binary package format for your software for the targeted platform (most users don't want to deal with make install and compiler output!) [ +50 points of FAIL ]
Your code requires GNU make?

Your code requires GCC?

Maybe these aren't fail points, but they bloody well should be.
Good criteria everyone shall follow, to protect himself