Wednesday, September 24, 2008

Talk: Reducing number of external repositories

At the Maemo Summit, I had a talk about the Extras repositories and the plans for the future. The slides can be found on Slideshare: The huge number of external repositories makes it increasingly hard to have a flawless experience for the user when it comes to package selection and installation. The proplems we currently face is:
  • Users are unable to easily find packages
  • Users experience conflicts between repositories
  • Unable to do quality testing on huge amount of repositories
  • – Is not able to hand pick packages from external repositories
One of the many things announced by Nokia was the creation of, where the best community made applications will be featured. I guess there will be more posts from Nokia on that topic later.

Back to the external repositories problem:
We started to retrieve all source packages from all know repositories found on Gronmayer. These packages are fed to the autobuilder to see if they would build. The status page looks like this:

Our plan to reduce the number of external repositories:
  • Build all source packages available in external repositories
  • Show status of builds
  • Try to discover missing sources (a lot of packages have no source available in the repo)
  • Help developers to fix build problems
  • Convince developers to move their packages into Extras
  • Convince teams inside Nokia to do the same
We hope to get visible results for this effort. Help with this effort would be very appreciated. Please try to convince developers to move their packages into Extras, when they haven't done so yet.

Together with Dave Neary, I did another talk. This one was about the history of the Maemo community and the changes we went through in the last few months. The slides for this talk are also available:


Varun said...

Wouldn't a workable alternative be something like providing a package in the Extras repository to add other repositories? That increases the discoverability of the other repos, even if that doesn't concentrate all the packages into a centralized repo.

Niels Breet said...

@varun: No, that would be the worst thing ever. The main point is that multiple repositories introduce dependency conflicts and other nasty issues. You really want less repositories instead of more.

We want to start doing testing on packages and their libraries, this can't be done on external repositories.

Slonopotamus said...

Man, I've requested access to extras-devel 2 weeks ago. And haven't received any reply since then. Of course I'll create my own repository, I don't have any other option.

Have you investigated why there are many repositories? Try getting access to extras, you'll understand the reason.

Niels Breet said...

@slonopotamus I'm sorry that I have missed your request. The holiday email backlog wasn't pretty.

Anyway, you should have received an invitation by now.

john said...

Why not create something like the new trend in "app stores", only they'd be free.

1) A repository that will appear in the App manager.

2) A group of known and reliable volunteers who will do varying levels of testing on the package and its contents to ensure that they're stable, and will grade their stability.

3) Once approved by the QA volunteers, a repository manager volunteer then puts it into the repo with feedback from the submitter and QA volunteers as to what category it belongs in, etc. (might also be an army of individuals who do this step, maybe with overlapping membership with the army of QA volunteers, but the person who performs this step should not be one of the people who did QA testing)

4) Anyone can submit a package for inclusion, it just has to pass the above testing, and then get put into the actual repo.

Niels Breet said...

@john We discussed this exact idea at the summit. The testing and QA would be placed between extras-devel and extras. Nokia also has plans for promoting the most excellent applications.

First step I will work on is to get a status page up, which shows the status for all packages. If there are problems with a package, they will be listed there. You will also have an easy way to report problems there.

More on that in an other blog post ;)

lusospikes said...

We have been talking about this for months. I would expect that by now we would have this already done. Google is investing on a "Google Android Apps" and Nokia should invest on this, otherwise, Maemo will be lagging behind the pack.

Joe said...

I'm eager to see a centralized repository as I've done the add-repo dance a number of times with the various major version upgrades; latest being the initfs-flash and kernel-flash updates that ate my dualboot.

I think I've run into ever issue you've listed, the latest confusion being the guessing game of which packages is really the latest build between two different repositories. Is it safe to install Chinook Extras packages if they don't exist in Diablo Extras and what are the implications? Why do I see updates in Chinook for versions that are actually newer then when I disable it, updates puts the older DiabloExtras packages back into my install again.

I can also see some of the more specialized repositories remaining seporate. MULLiNER is gracious enough to offer his personal packages and they are pretty specific so I wouldn't push that towards Diablo-Tools too hard. Elkins and Debfarm though?

I don't think half the potential in this hardware has been explored. Between the community project for a NIC kernel modual (please, include NAT and make it come out of monitor mode cleanly) and possability of getting full host mode USB out of it like Chinook used to do, there are some very exciting areas to explore. Sadly, my expertise are on the OS use and auditing side rather than development and cross compiling otherwise I'd be very happy to help out more than just being a mouth and premoter.

I've looked at Android and the iPhone and Maemo is already so much further ahead. I just hope it can survive the Cult of Mac and Google's brand strength. Consolidating the repositories an getting on with exploring what we can do to this hardware through the software is a big part of that survival.