2013-05-30

Tycho and pre-p2 update sites

(from "Simplifying The p2 Process, Part 3: Associate Sites")
Even as p2 has taken over the world domination when it comes to Eclipse update sites, you can still find old-style update site around. This really doesn't matter as long as you want to install features in Eclipse itself - it just takes longer, as all features of the update site is downloaded - but if you want to use these old-style update sites in a Tycho build, then you have a problem as Tycho only support p2 based repositories, not the old-style update sites.

During the development the next version of our AGETOR product, we recently had this exact problem with the Elver update site, which contains the needed plug-ins if you want to work with Teneo. In this post, I'll try to describe how you can solve the problem, and likely also get a faster and more stable module resolution with your Tycho build.

Why does Tycho not support old-style update sites?


Well, basically the old-style update sites are based on a site.xml file rather than the content.xml and artifacts.xml files of a p2 repository. The site.xml file just contains a list of the features of the update site, but no information about the plug-ins of the features or the imports and exports of these. The later information is exactly what you find in the content.xml file of p2. When you install new features in Eclipse, the resolution is based only on plug-ins as described in features, whereas the resolution for building is also based on imports and exports and sometimes Java packages. This means, that by downloading features and using the information in these, it is possible to install features, but that is not enough to resolve the dependencies when compiling a bundle.

Tycho and Target Platforms


When you build a product with Tycho, you need a target platform to describe the features and bundles that are usable in the product. There are several different ways to declare the target platform in Tycho (see the wiki page on Target Platforms), but most often you will end up with a target definition file which references the used p2 repositories and update sites.

There are two different problems with this approach:
  • network problems and long response network times will directly influence your build times, which can be rather irritating when nothing changes in the referenced repositories and update sites...
  • and, if anything does change in the referenced repositories or update sites, it can give some rather surprising resolution errors in a build that previously didn't have any problems.

Internal Repository


In order to avoid these problems, you often also generate a new internal repository with all the needed features and plug-ins and let all your target platform definition files point to this internal repository.

You don't have to re-generate the internal repository very often (we do it once a month and check the differences), and you can generate the internal repository directly in Tycho with the mirror task of the addition tools (see below). Lastly, you can filter out old versions of features and special problematic units from the repository - getting both a faster and more stable repository - so all-in-all there are no reasons not to do this.

Old-Style Update Sites


If the target platform includes any old-style update sites, then the generated internal repository will work just fine in Eclipse, but not so in your Tycho build as the new site will not include all the dependency information needed by Tycho as described above. It would have been nice - very nice - if the mirror task and p2 would just include these dependencies automatically, but that is not so (yet). Instead, the mirrored information from old-style update sites will be marked as being partial and the newest version of Tycho (0.18.0) will now complain where the older versions will silently ignore the problem.

But everything is not lost, as p2 includes a tools to read through the features and bundles of an old-style update site and generate the content.xml from this - the original update site just have to be present in the local file system.

The Solution


So the solution is to first download and convert the old-style update sites into proper p2 update site - using the publish-features-and-bundles task of the additional tools - and then use these in the generation of the internal repository.

This all boils down to the following POM that can be used to generate a new internal repository. It usually takes 6-8 hours with our list of external repositories, but that is fine...

I have added some comments after each section to explain the important features.

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <artifactId>com.agetor.target.gen</artifactId>
  <version>4-SNAPSHOT</version> <!-- <<<< -->
  <packaging>pom</packaging>

We number our generated internal update site, so we easily can get back to a previous version. Current version is number 4.

Please note that this module is not part of the normal build reactor of the product. We just build this module manually once per month to check that nothing significant has changed in the many repositories and update sites we consume.

  <parent>
    <artifactId>com.agetor.parent</artifactId>
    <groupId>com.agetor.v5</groupId>
    <version>1-SNAPSHOT</version>
    <relativePath>../com.agetor.parent</relativePath>
  </parent>

We only use the parent site to pick up Tycho repositories and dependencies... But we could just as well have added this information directly to this POM...

  <build>
    <plugins>
      <plugin>
        <groupId>org.eclipse.tycho.extras</groupId>
        <artifactId>tycho-p2-extras-plugin</artifactId>
        <version>${tycho.version}</version>

        <configuration>
          <publishArtifacts>true</publishArtifacts> <!-- <<<< -->
          <!-- Include pack200 artifacts -->
          <includePacked>true</includePacked> <!-- <<<< -->
          <reusePack200Files>true</reusePack200Files> <!-- <<<< -->
        </configuration>
The common configuration for the additional tools: we want to copy the artifacts in the mirror operations and we want both the raw bundles and the pack2000 versions.

        <executions>
          <execution>
            <!-- Fetch and merge all old-style update sites into 
              a single "partial" p2 repository -->
            <id>pre-p2-content</id>
            <phase>package</phase>
            <goals>
              <goal>mirror</goal> <!-- <<<< -->
            </goals>
            <configuration>
              <append>false</append>
              <latestVersionOnly>false</latestVersionOnly>
              <!-- The destination directory to mirror to. -->
              <destination>${project.build.directory}/old-orig-repository</destination>

              <source>
                <repository>
                  <url>http://www.elver.org/eclipse/2.0.0/update/</url> <!-- <<<< -->
                  <layout>p2</layout>
                </repository>
              </source>
            </configuration>
          </execution>
First we must download all the old-style update sites to the local file system. In this case, there are just one update site to copy, but additional sites can be added above to copy all the sites to the same local folder.

          <execution>
            <!-- Generate proper p2 site for the old style update sites -->
            <id>generate-p2-content</id>
            <phase>package</phase>
            <goals>
              <goal>publish-features-and-bundles</goal> <!-- <<<< -->
            </goals>
            <configuration>
              <append>false</append>
              <latestVersionOnly>false</latestVersionOnly>
              <artifactRepositoryLocation>${project.build.directory}/old-p2-repository</artifactRepositoryLocation>
              <metadataRepositoryLocation>${project.build.directory}/old-p2-repository</metadataRepositoryLocation>
              <sourceLocation>${project.build.directory}/old-orig-repository</sourceLocation>
            </configuration>
          </execution>

Then we generate the needed content.xml and artifacts.xml files. The operation will automatically copy the features and plug-ins to the new repository.

          <execution>
            <!-- Mirror release and milestone repositories. We only include the 
              newest versions from the milestone repositories -->
            <id>milestone-content</id>
            <phase>package</phase>
            <goals>
              <goal>mirror</goal> <!-- <<<< -->
            </goals>
            <configuration>
              <append>true</append>
 <!-- <<<< -->              <source>
                <repository>
                  <url>http://download.eclipse.org/technology/nebula/snapshot</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>http://tonnymadsen.github.com/ui-bindings/updates/release/</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>file:///${project.build.directory}/old-p2-repository</url> <!-- <<<< -->
                  <layout>p2</layout>
                </repository>
                <!-- same as release-content -->
                <repository>
                  <url>http://download.eclipse.org/tools/orbit/downloads/drops/R20130517111416/repository/</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>http://download.eclipse.org/eclipse/updates/3.8/</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>http://download.eclipse.org/rt/rap/2.0</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>http://download.eclipse.org/releases/juno</url>
                  <layout>p2</layout>
                </repository>
              </source>
            </configuration>
          </execution>
Then we copy the latest version of all the features and plug-ins from all the repositories. Note that we have added the repository we have just generated in the previous step.

          <execution>
            <!-- We include all release content, even if a newer version exists 
              in the milestone repositories -->
            <id>release-content</id>
            <phase>package</phase>
            <goals>
              <goal>mirror</goal>
            </goals>
            <configuration>
              <append>true</append>
              <latestVersionOnly>false</latestVersionOnly> <!-- <<<< -->
              <source>
                <!-- source repositories to mirror from -->
                <repository>
                  <url>http://download.eclipse.org/tools/orbit/downloads/drops/R20130517111416/repository/</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>http://download.eclipse.org/eclipse/updates/3.8/</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>http://download.eclipse.org/rt/rap/2.0</url>
                  <layout>p2</layout>
                </repository>
                <repository>
                  <url>http://download.eclipse.org/releases/juno</url>
                  <layout>p2</layout>
                </repository>
              </source>
            </configuration>
          </execution>
        </executions>
      </plugin>

Last, for all the Eclipse release repositories, we must include all versions from the repositories as many of our modules can otherwise not be resolved. E.g the newest version of many Eclipse specific bundles is 4.2, but we needed version 3.8 as well.

    </plugins>
  </build>
</project>

That's it... It does take some time, but it will generate a new repository that can then be copied to an internal web server and be referenced in the target definition files.
Posted by at
Labels: ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)