Maemo 5 Extras reaches 3.5M downloads

The maemo.org Extras repository for community contributed software has reached 3.5 million downloads for Maemo 5. Software is offered through maemo.org free of charge and the repository is open to all developers. maemo.org Downloads is the web interface to the software available through maemo.org Extras.

We see the number of downloads grow every day. This makes the maemo.org Extras repository a great place for developers to get their software in the hands of end users. If you are interested in publishing your software through maemo.org, please visit the wiki.

We introduced improved download statics to show developers exactly how many downloads they received. The FM Radio Player application has received over 138000 downloads. FM Radio stats below:

The Extras repository has only been enabled on N900 devices by default in a recent software update. Before this update users needed to manually activate the repository in their Application Manager's catalog list. Also keep in mind that the Application Manager is not very easy to find, because of it being put in the 'More' menu.

Overall I think the number is quite impressive as these devices aren't available for a long time and a lot of people had problems even finding a device in their country.

Extras dput and promotion changes

With the change to the new server infrastructure, we also changed the way developers can use dput to upload their source packages to the autobuilder. We created a special host named drop.maemo.org for uploads using dput (and scp). The uploading to Extras wiki page has been updated with the correct information. Short story: replace garage.maemo.org with drop.maemo.org and continue like before.

Another change we had to make was how packages are promoted from diablo Extras-devel to Extras. The old promoter was no longer suitable for the current setup and needed to go. Promotion now works the same way as it works for fremantle Extras-devel to Extras-testing (direct promotion). The package interface for diablo Extras-devel can be found here.

There are still a few features missing for the diablo part of the packages interface, but promotion should work. I'll add build logs and context sensitive searches at a later point.

Backwards compatibility broken PR1.1 SDK

I've been discussing this issue with some people before as hypothetical case, but now it seems that we run into it: Compiling and application against the PR1.1 SDK creates packages which can not be installed on earlier firmware releases.

In this case we have have a libosso version which is higher than the one in previous releases. As this dependency gets automatically added when compiling in the PR1.1 SDK this poses a problem.

The autobuilder uses the repository.maemo.org repository, so it automatically uses newer packages when they are available.

For Extras this means that install of an application which is compiled against the new SDK fails without any description we can expect an end-user to understand. This is something which should be prevented.

How can we work around this problem:

1: Only compile against the original SDK.

This prevents new features from ever be available to developers, but should work until there is real API/ABI breakage in a new firmware.

2: Use version specific repositories

This needs Application Manager support as we need to fetch from a separate repository every time. Also requires us to build against every sdk version known to man.

3: Depend on >= mp-fremantle-generic-pr | maemo-version

We would need a hack in the autobuilder to add depends to pr and maemo version. This way a user needs to upgrade to at least the required firmware image. I think this will make it easier for an end-user to understand what is happening.

We could, with help of the AM team, even detect in the AM that a firmware upgrade is required and give a the end user a nice warning/description.

Currently the AM doesn't have any means to detect which firmware version a package requires. Option 3 solve that issue at the same time.

If you have an alternative solution on how to go about fixing this
issue, then please let me know.

Discussions on the maemo-developers list or talk.

Maemo 5 Downloads now available

For Maemo 5 applications which managed to reach the Extras repository through the community QA testing queue, maemo.org Downloads now has a home.

The Download catalog automatically imports all information for applications in Fremantle Extras. The only thing you need to add is a nice screenshot. (Or more if you want.)

There are quite some applications in the QA testing queue which are almost ready for promotion to Extras. We should see more entries after the Summit.

If you are coming to the Maemo Summit in Amsterdam, see you there!

Proposal: XSBC-Bugtracker in debian package

While implementing the maemo.org packages interface, I wanted to add a link to the bugtracker for a package or project. Eventually integrating bug reporting/searching, but more on that later. As all information is gathered from the debian packages themselves, it would make sense to have the bugtracker URL available in the control file.

There doesn't seem to be a default field in debian control files for and URL of the bugtracker which the project is using. Therefor I want to propose introducing the XSBC-Bugtracker field.

XSBC-Bugtracker:
The URL of the bugtracker for this package, preferably (when applicable) the URL where you can directly file a bug. The content of this field is a simple URL without any surrounding characters such as <>.

Example:

XSBC-Bugtracker: https://bugs.maemo.org/enter_bug.cgi?product=Wormux

By putting the bugtracker URL inside the package, people always have a way to find out where to file bugs. This could even be used for filing bugs through an application interface or crash reporter in the future.

When this feature is accepted, it should be added to the Packaging Policy too.

Please let me know what you think. Discussion at talk.maemo.org.

Fremantle maemo-select-menu-location change

While working on the maemo.org package interface, I noticed that quite a few package's developers didn't notice the changes in Fremantle regarding maemo-select-menu-location.

Fremantle changes the way applications are visible in the menu. There are no sub menus like "Extra" or "Utility" anymore. All installed 3rd party software will be visible under "Applications" by default.

This means that maemo-select-menu-location is obsolete and has been removed from the SDK. More info about the removal can be found in this bug.

There are currently about 60 package instances in Fremantle Extras-devel that depend or pre-depend on this obsolete package. These packages can not be promoted to Extras-testing until their author updates the package.

Daniel has started working on a Porting to Fremantle QA wiki page, which should point out some changes needed when porting to Fremantle. I hope this will help developers with the problems they encounter for now.

Introducing maemo.org packages

Starting with Fremantle, packages can now be managed by developers through the maemo.org packages interface. Community testers can help in the QA process by testing applications in the queue and giving thumbs up/down for a package.

At the same time we open the extras-testing and extras repositories for Fremantle for promotion through the interface.

The interface has the following features:

• Package search
• Package listings per repository
  
• Only allow maintainers to promote a package
• Request package maintainership
• Automatic package maintainership assignment on first upload of a package
• Automatic dependency tree checking
• QA queue for testing repository
• Promotion unlock to stable (Extras) repository based on karma score
• Leaving comments when you encountered problems with a package

We index all Fremantle Extras*, SDK and Nokia-binaries repositories, so we can check for availability of every dependency. A warning will be shown on a package information page, when there are missing dependencies. Package promotion will be blocked until the issue has been resolved.

At this moment the interface is in beta testing mode, so please be aware that there might be some rough edges. Promotion will happen manually for the first few promotions, so we can make sure that everything is working. Feel free to file bugs in bugzilla, if you encounter any.

Improvements will come in the next few weeks. More integration with the autobuilder is already coming next week.

In a later post, I will discuss the new promotion process for Fremantle a bit more in depth. This is still something that needs to be refined a bit.

Improving maemo.org Downloads

As part of my tasks in the January 2009 Sprint for maemo.org, I want to gather improvement ideas for downloads.maemo.org. A list of improvements that I have come up with can be found at the wiki page. If you have ideas on how we can make Downloads work better for you or have some cool feature you want to have added, please add them to this wiki page.

At the end of this Sprint, I will pick some of the ideas to which I will commit for the next Sprint. Some items that have been added to the wiki page already are:

Category reorg

We had long discussions about what packages categories to use. Implementation and deployment is currently blocked by bug #1805. The current catalog still shows a mostly randomly-selected list of categories. It should be reorganized to use the official list of categories agreed on by the community.

OS200x vs Maemo x

In the past Nokia referred to OS releases OS2007, OS2008, etc. The use of OS200x has been deprecated in favor of Maemo x (e.g., Fremantle is Maemo 5). The catalog is currently organized by OS200x, this needs to be changed to the new versioning scheme.

Automatically create entries in Downloads when package in Extras has user/* section

At the moment not all applications which are available in Extras are listed in Downloads. A script could gather basic information for all user applications in Extras and create entries in the catalog. These entries can later be updated by users to give more details about the application.

Add Application Manager install failures feedback to downloads

There is a plan to integrate install status feedback into the Application Manager. Downloads could use statistics from installation failures to display the "quality" of a package.

Packaging Policy change proposal

After long and heated discussions, we finally managed to finish the Better Package Categories task. When I started the discussion in April, I knew it would not be an easy task. It remained on my task lists over a lot of sprints, but now we have a result.

Ryan posted about this and proposed future in his blog.

"So, what's next?"

Now we need to get these proposed changes accepted into the Packaging Policy, so we can start to apply them. I created a wiki page for the proposed changes to the policy. Please see if you would like to see the wording changed. I will push this task for the November 2008 Sprint.

The autobuilder and Extras Assistant will soon start to give warnings when your package is not using one of the official categories. These warnings will not prevent you from uploading your package, but please try to update your package to make it comply with the policy.

As we are going to change the policy, the wiki page might also be a good place to propose other changes you would like to see in the document.

New features autobuilder and Extras Assistant

Rejecting packages when the same version is uploaded for the second time.

The autobuilder has been modified to reject a package when the same version is already available in the extras-devel repository. This change will force developers to increase their package version number after each successful upload and build. In the past we've had problems in the repository where a package was uploaded twice and caused 'Size Mismatch' errors.

Package signing no longer required.

The autobuilder and Extras Assistant no longer require packages to be GPG signed. It seemed to cause a lot of grief for developers without any real benefits. This change is targetted to making uploading packages easier. Every uploader has to be authenticated to upload a package, so we can already trace back the uploader. The autobuilder signs packages which are moved into the repository.

Upload to multiple repositories at the same time.

You can now use the Extras Assistant to upload a source package to chinook and diablo at the same time. No need to do two separate uploads anymore. This feature has been requested for quite some time, let's hope it helps.

Reducing number of external repositories #1

In my talk at the Maemo summit 2008, I talked about starting an effort to reduce the number of external repositories. Talking about this helps, but actively contacting developers and repository owners helps even more. This week I started by creating a list of all repositories we need to target. When this was done, I started to contact a lot of developers to convince them to move their packages into Extras and then close their repository.

A funny thing to mention is: Nobody had any objections to this.

Most responses were apologies for not doing it sooner. When I started this, I expected to have some resistance, but so far there was none.

This is a good thing! It seems the community wants things to change too.

How can you help?

If you know somebody developing applications in their own repository, try to convince them to move their packages into Extras(-devel). Please edit the wiki page to reflect the status of each repository.

With the help of a few people, I think we can get number of repositories in the list down quickly!

Visible results

When looking at the Gronmayer listing, you can see that more and more repositories are being removed and end up as: 'Repository is offline'.

Talk: Reducing number of external repositories

At the Maemo Summit, I had a talk about the Extras repositories and the plans for the future. The slides can be found on Slideshare: The huge number of external repositories makes it increasingly hard to have a flawless experience for the user when it comes to package selection and installation. The proplems we currently face is:

• Users are unable to easily find packages
• Users experience conflicts between repositories
• Unable to do quality testing on huge amount of repositories
• maemo.nokia.com – Is not able to hand pick packages from external repositories
  

One of the many things announced by Nokia was the creation of maemo.nokia.com, where the best community made applications will be featured. I guess there will be more posts from Nokia on that topic later.

Back to the external repositories problem:
We started to retrieve all source packages from all know repositories found on Gronmayer. These packages are fed to the autobuilder to see if they would build. The status page looks like this:

Our plan to reduce the number of external repositories:

• Build all source packages available in external repositories
• Show status of builds
  
• Try to discover missing sources (a lot of packages have no source available in the repo)
  
• Help developers to fix build problems
• Convince developers to move their packages into Extras
• Convince teams inside Nokia to do the same
  

We hope to get visible results for this effort. Help with this effort would be very appreciated. Please try to convince developers to move their packages into Extras, when they haven't done so yet.

Together with Dave Neary, I did another talk. This one was about the history of the Maemo community and the changes we went through in the last few months. The slides for this talk are also available:

Maemo Summit 2008, Awesome!

The gathering of everybody Maemo has ended yesterday. 2 days filled with very interesting or even amazing meetings, talks and gossip. There was a very sharp contrast with the days before that at OSiM. I want to bet that the Maemo community was able to achieve more at the summit than all projects at OSiM together.

I've met a lot of people and it was very good to see the faces behind the email addresses or IRC nicks. We should not forget the benefit of meeting people in real life. Going to a bar or a restaurant together to have a nice dinner and have some German beers even adds more to this of course.

I'm glad that some of our rockstar hackers had a chance to meet the Nokia people involved in their favourite subjects. A lot of the explanations, about why something was decided this way, are better explained in a personal conversation.

Special props for lardman doing a talk on dsp programming in 30 minutes. The amount of information he managed to put in one talk was mind boggling.

As a maemo.org guy, I have had the opportunity to discuss a many plans and this will likely be more than enough work for me for the coming 10 years ;) Ed and Sasha(Wii tennis king!) had some valuable ideas for managing the repositories and reporting problems. I really hope we can implement those as soon as possible.

Next blog post will be about my talks and I will make the slides available there.

See you at a next meeting?

[Edit: Somehow this didn't get posted while I was at Tegel Airport yesterday]

maemo.org/downloads automatic updates from Extras

The maemo.org downloads section now automatically picks up updated versions from the Maemo Extras repository. The 'Fresh' list will show the applications that were recently uploaded or promoted to Extras. Developers used to need to update their application entry themselves. By doing the updating automatically, users should not see outdated information about applications.

For the automatic update to work your application needs to be:

1. Available in Maemo Extras
2. Available in the maemo.org downloads section where the 'Project ID' equals the name of your debian package.

Package updates are fetched from the most recent repository for each OS we support. Diablo for OS2008, Bora for OS2007 and gregale for OS2006.

One enhancement I would like to add is automatically update the 'Changes in latest version' field for the entry in downloads. I would like comments from the community on how developers should supply this information.

One option would be to fetch it from the changelog. Problems here are that there aren't many packages using a changelog at the moment and we would need to filter out the real changes from the packaging revision updates.

Another option would be to let the developer enter this data while promoting the package to extras. We could add this step to the promotion interface.

Comments are appreciated.

Application in Extras-Devel OK? Promote it!

Now that we have the autobuilder in place for Diablo, it is a good time to talk about how to get your package from Extras-Devel into the Maemo Extras repository.

Before a package will end up in Extras and is installable for end-users on their device, it will go through a few steps:

1. source package uploaded to autobuilder incoming queue
2. autobuilder builds package and moves it into the incoming queue from Extras-Devel
3. queuemanager for Extras-Devel will put the package in the Extras-Devel repository
4. developer checks packages from Extras-Devel and if the package is OK:
5. promote the package to Extras using the Promoter.

Step 5 is the step where you can use the Promoter to promote your package. Make sure you test your application while it is in Extras-Devel, so you don't promote a broken package. After promotion and a bit of patience, your package will show up in Extras and everybody will be able to download it.

We have two Promoters is place:

• Diablo Promoter
• Chinook Promoter
  

The Promoter is a community project on garage.maemo.org. The code can be found in the svn repository. If you have any suggestions on how to improve the interface or would like to add functionality, please propose and discuss it on the maemo-developers mailinglist.

We are still working on documenting Extras and Uploading to extras in the wiki and of course work towards The Big Plan. Feel free to edit the wiki and improve the articles, this is a community project.

Let's work together to get more applications available in Maemo Extras!

Rebuild all chinook source packages on autobuilder

Currently Ed Bartosh and I are working on taking all source packages from the extras repository and try to build them on the autobuilder. The goal of this chinook rebuild effort is to get a set of packages buildable 'from scratch'.

Once we are able to build all(most?) packages on chinook, we can try to automatically build them for diablo. So we can have a lot of packages in the diablo repository at release of diablo.

There is a web page up with the first run, which was done over the weekend. All packages are listed in build order, based on dependencies, with their build results for i386 and armel. If a package build failed, a link to the build log is provided.

If you provide a source package in extras, please check if your package is building OK. If you only provide binary packages in extras, we would like to encourage you to provide source packages too! We could really use some help from the community in pushing towards 100% OK build of every package in extras.

When you have updated your package to fix the issues, please use the 'request rebuild' option on the packages list. This gives us the opportunity to track changes as a result of the list. We will rebuild packages on a regular interval and post a summary to maemo-developers. Let's see if we can get all packages to build!

Autobuilder for maemo extras repository part 2

Some time has passed since we first announced the maemo autobuilder for the extras repositories. Some people have tested it and we got a little bit of feedback. Not as much as we hoped, but I'm sure that is going to get better shortly.

Ed has been working on making the autobuilder more user-friendly. You will now receive a status message per mail when your package has been processed. If there were any errors (md5 sum failed, files missing, wrong pgp key or package not signed), you will find that out in the email.

Some benefits of using the maemo autobuilder:

• Source packages always available in the repository
• i386 packages are automatically added to the repository, so you can use them in the SDK.
• Prevent obviously broken packages from entering the repositories.
  

I have been improving the assistant by adding more checking and fixing more bugs. It turned out that not everybody was able to request rights to upload packages to the repository. Sorry about that!

I would like to ask all developers to try out the autobuilder and assistant, so we can be in a good shape before Diablo gets released. We can only move forward if you get involved!

Feel free to contact me if you need any help or information.

maemo packaging policy draft

A draft for the maemo packaging policy has been released for comments by the community.

What is the maemo packaging policy good for?

Like it's Debian counter part, the maemo policy specifies how packages should be built. It will exactly tell you what your package should implement to be a proper maemo package.

The maemo packaging policy is essentially the same as the Debian policy, but there are some changes especially for the tablets as opposed to desktops.

What good does it do me?

Well, not a lot at the moment. But it can soon! We (the maemo community) need to agree on this policy and see how we can implement it.

We want to setup a wiki page where you can annotate the policy draft and talk about things you think should change. This will be done soon, I hope.

Until then, please discuss the topic on the maemo-developers list.

Your input is welcomed.

Autobuilder for extras repository public beta

A while ago we discussed the idea of an autobuilder for the extras repository. We have been working on creating such an autobuilder and have been testing it in private. As a result we now think it is time to do a tests with a larger group and make it publicly known. Please give it a try and tell us your results.

- What does the autobuilder do?

The autobuilder takes a source package from the incoming queue and tries to build it in a fresh environment. The builder fetches all dependencies from the extras-devel repository. If it can't find the dependency in extras-devel, it will fail and send a mail to the extras-cauldron-builds list.

If there are no problems with the package, the builder can create armel, i386 and source packages. Those will be put in the extras-devel repository after a successful build.

Instructions for the builder can be found at here.

The builder only handles building one package at a time. If you have dependencies that are not present in the extras-devel repository, please upload these first.

- web-based assistant

We have also created a web-based assistant to help you with requesting rights to upload and creating/uploading of source packages. You can upload packages to the builder with either dput or the assistant.

- How do I get my package into extras?

After a successful build a package will appear in the extras-devel repository. You can move your packages from extras-devel to extras with the promotion interface. Instructions for the promotion interface can be found here.

We would like to invite all developers to take a look at the autobuilder and try it out. Please discuss problems and feature requests on the maemo-developers list.

More information about our effort can be found at the extras-cauldron website.

[RFC] Maemo package guidelines: mandatory categories

Here is my first suggestion to clean up the complete mess we have at the moment when it comes to package categories in the maemo extras repository. There is no official list of categories, which has brought us to state we are in now.

We have these nice categories for example: 'Boingo', 'Canola'. Those should never be a category by themselves. We also have a lot of duplicates like 'cli' ,'Commandline' and 'Web','www' and 'Utilities','utils'.

This really has to stop as this is confusing for end users. We, the maemo community, need to find a solution and fix this.

If we look at Debian, we can see that they have the following list of categories:

admin, base, comm, contrib, devel, doc, editors, electronics, embedded, games, gnome, graphics, hamradio, interpreters, kde, libs, libdevel, mail, math, misc, net, news, non-free, oldlibs, otherosfs, perl, python, science, shells, sound, tex, text, utils, web, x11

My suggestion would be to base our list off the Debian list and remove the categories that are not suitable for Maemo. We might also want to add some categories if we find some missing.

admin, comm, devel, doc, editors, games, graphics, interpreters, mail, net, news, utils

and add:

desktop, database, education, internet, multimedia, office, scientific, security, system, travel

Please feel free to suggest other categories. Try to keep them as broad as possible. I would really like to get a list of categories where every application can be in at least one category. It would be nice not to need the 'misc' or 'other' category.

Perhaps it would also be a good idea to have the Application Manager display the pretty name for each category. e.g. comm -> Communication. That might be step 2 though.

I also would like your feedback on this idea:
"For diablo we only accept packages in the extras/extras-devel repositories when they have a valid category."

I'm really not sure if we can do this in time for diablo, but at least we can try to get the community to agree on this. I don't think we can do anything for existing repositories, but at least we could try for the new ones.

Please respond with your ideas in the comments section, but keep it to the category subject only.

Edit:

There seems to be a list of categories for the Application Manager. I don't think that list has enough categories, but it is a start.