Fix Broken Packages

26 Sep 2019 Linux tips shell package_management

A server machine failed at upgrading from Ubuntu14.04 to Ubuntu16.04, and many packages are left in half-installed status, even apt is not working. I struggled several days to fix those broken packages, and this post documents some tricks I learnt.

Fix Broken apt

There is chance for apt, the commandline package manager, to stop working for some reasons. I encountered three times: two for the Glibc version mismatch and one for header signature.

First time is because apt itself has been upgrade to higher version in the kernel upgrade, and /etc/apt/sources.list has been replaced to use Xenial (for Ubuntu16.04). The higher version of apt will try to look for higher versions of Glibc shared library to execute. I replace the Xenial in /etc/apt/sources.list with Trusty (for Ubuntu14.04) because I realized some tools only work with old linux kernel (3.19). I download (from Launchpad Trusty repo) and install apt and related .deb packages by dpkg.

Some basic dpkg usages:

# install a .deb package
$ sudo dpkg -i <path to .deb file>
# list installed packages, better used with grep
$ dpkg --list
# remove an installed package
$ sudo dpkg -r <package name>
# allow temporarily break the dependencies in case dead loop
$ sudo dpkg --ignore-depends=<pkgs depend on the one to remove> -r <pkg name>
# configure package to fully function (list will show `iU` instand of `ii`)
$ sudo dpkg --configure <package name>

After the apt is downgraded, it can perform most of its functionalities without issue. While I failed to use it to downgrade perl. Unlike dpkg does the package maintenance in a direct way, apt will invoke multiple scripts for pre-processing and post-processing. Some of the scripts rely on perl, but perl needs to be downgraded now to work with lower version of Glibc. The solution is again, using dpkg to install perl and related packages to replace the newer version in use.

The last problem raised when apt has a script triggered to process a package while the script failed on format checking (I suspect this is a new feature added by Ubuntu16.04 for security), and the solution found online is just to add a signature-like header.

Deal With Dependencies

Once apt back to work, you can use it to downgrade packages by install packages with versions specified, like this:

sudo apt install landscape-common=14.12-0ubuntu6.14.04

While it may stop due to unmet dependencies. In most of my cases, the dependency issue is because the package I want to install depends on a specific lower version of other packages, while the lower version is not going to be installed because a higher version exists. The solution is to downgrade a chain of packages all together in one apt command. It also could be the dependent package has conflict with alternative packages installed (e.g., libparted0 conflicts with libparted2) or will break other packages that depends on the package to be replaced (e.g., libparted-fs-resize0 depends on libparted2). There are commands to help you to resolve those dependencies:

apt-cache depends <package name>
apt-cache rdepends <package name>

For those packages needed to be removed or replaced together when install a package, there is a easy trick introduced in the article:

# install pkg1 meanwhile remove pkg2 due to conflict or broken dependency
sudo apt-get install pkg1 pkg2-
# remove pkg2 meanwhile install pkg1 to replace its support for other packages
sudo apt-get remove pkg1+ pkg2

With above tricks learnt, I need to composite a really long apt command to correct the versions of packages all that once. Given 1500+ packages installed, it is time-consuming to go over all the packages manually. Here are couple commands helped me to get the list of all the version options for all the packages installed.

# to get a list of installed packages (header removed by tail command)
dpkg --list | tail -n +6 > pkgs.txt
# to truncate the leading status, following descriptions, and arch version
sed -i -e 's/..\s\+\(\S\+\).\+/\1/' -e 's/:.\+$//' pkgs.txt 
# query versions available in the source repo with input from file line by line
while IFS= read -r pkg; do echo "$pkg"; apt-cache showsrc "$pkg" | grep '^Version'; done < pkgs.txt | tee vers.txt
# prefix version with '=' in favor of making apt command
sed -i -e 's/Version: /=/' vers.txt

Because there are packages having multiple versions, and the latest version may not be listed as the first one or the last one all the time, we need to make an integlligent sellection. It will take at least 2 hours for me to go through the version list, so I instead spent the 2 hours on writing a python script to help.

Basically, the script extracts the numbers from the version strings, and append the one having the larger number earlier to package. But in the case having date like 20190926 exists in the version numbers, it prompts to ask user to make a decision. Because many those packages have similar version options, so the script would memorize the decision made by user to reduce asking. This script saved a lot of my time to composite the long long apt command.

Remove Kernel Upgrade Notification

To prevent other users try to upgrade kernel and cause problem again, better to remove the notification message shown upon login. The message is generated by the configure files under folder /etc/update-motd.d, so renaming or taking away the execution permission could achieve my purpose.

Credits