[[email protected]:~]$

Getting to know RPM


This will be a brief, no frills, overview of RPM and what it is. There will be a few examples, followed by scenarios with issues you may come across frequently. This should be a sufficient introduction to get you comfortable with rpm. Note this will not include installing and uninstalling packages, as there are a million guides on that. Rather we will touch on some of the other basics uses and features that are, at times, overlooked.

Index

  1. RPM
  2. RPM Keys
  3. Useful Commands
  4. Common Issues

RPM

RPM - (Redhat Package Manager) A tool developed by redhat for rpm based systems. The tool is not available for debian based such as ubuntu. The main issue with rpm, despite it’s many uses, is dependency hell. RPM does not have an efficient way of tracking dependencies like yum or apt do.


RPM Keys

These are ‘fake packages’ created to store and manage keys. If you erase/remove them, you will be unable to update/upgrade any package from that repo.

List all installed gpg-keys for packages.

[[email protected] ~]# rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-ef8d349f-57b6233e    gpg(Puppet, Inc. Release Key (Puppet, Inc. Release Key) <[email protected]>)
gpg-pubkey-352c64e5-52ae6884    gpg(Fedora EPEL (7) <[email protected]>)
gpg-pubkey-f4a80eb5-53a7ff4b    gpg(CentOS-7 Key (CentOS 7 Official Signing Key) <[email protected]

Remove a key, in this case, we removed the one for puppet.

[[email protected] ~]# rpm -e gpg-pubkey-ef8d349f-57b6233e
[[email protected] ~]# rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-352c64e5-52ae6884    gpg(Fedora EPEL (7) <[email protected]>)
gpg-pubkey-f4a80eb5-53a7ff4b    gpg(CentOS-7 Key (CentOS 7 Official Signing Key) <[email protected]>)

Add a key, in this case, we re-added the key for puppet.

[[email protected] ~]# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-puppet5-release
[[email protected] ~]# rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE}\t%{SUMMARY}\n'
gpg-pubkey-352c64e5-52ae6884    gpg(Fedora EPEL (7) <[email protected]>)
gpg-pubkey-f4a80eb5-53a7ff4b    gpg(CentOS-7 Key (CentOS 7 Official Signing Key) <[email protected]>)
gpg-pubkey-ef8d349f-57b6233e    gpg(Puppet, Inc. Release Key (Puppet, Inc. Release Key) <[email protected]>)

Useful Commands

Query all packages. This will return a list of all packages on the system. This can be pared with a package name to query all packages with that name. If you pair it with a package the a flag is not needed.

[[email protected] ~]# rpm -qa kernel
kernel-3.10.0-862.11.6.el7.x86_64
kernel-3.10.0-862.14.4.el7.x86_64

Find when a package was last updated.

[[email protected] ~]# rpm -q --last kernel
kernel-3.10.0-862.14.4.el7.x86_64             Tue 23 Oct 2018 01:33:20 AM UTC
kernel-3.10.0-862.11.6.el7.x86_64             Sat 15 Sep 2018 01:30:42 PM UT

Find out which package owns a particular file.

[[email protected] ~]# rpm -qf /etc/plymouth/plymouthd.conf
plymouth-0.8.9-0.31.20140113.el7.centos.x86_64

View the changelog of a package. This is also helpful in finding out if a previous CVE was addressed.

[[email protected] ~]# rpm -q --changelog kernel
* Tue Aug 14 2018 CentOS Sources <[email protected]> - 3.10.0-862.11.6.el7
- Apply debranding changes
 
* Fri Aug 10 2018 Jan Stancek <[email protected]> [3.10.0-862.11.6.el7]
- [kernel] cpu/hotplug: Fix 'online' sysfs entry with 'nosmt' (Josh Poimboeuf) [1593383 1593384] {CVE-2018-3620}
 
* Thu Aug 09 2018 Frantisek Hrbata <[email protected]> [3.10.0-862.11.5.el7]
- [kernel] cpu/hotplug: Enable 'nosmt' as late as possible (Josh Poimboeuf) [1593383 1593384] {CVE-2018-3620}
[...]

Find the documentation for a package.

[[email protected] ~]# rpm -q --docfiles openssh
/usr/share/doc/openssh-7.4p1/CREDITS
/usr/share/doc/openssh-7.4p1/ChangeLog
/usr/share/doc/openssh-7.4p1/INSTALL
/usr/share/doc/openssh-7.4p1/OVERVIEW
/usr/share/doc/openssh-7.4p1/PROTOCOL
/usr/share/doc/openssh-7.4p1/PROTOCOL.agent
/usr/share/doc/openssh-7.4p1/PROTOCOL.certkeys
/usr/share/doc/openssh-7.4p1/PROTOCOL.chacha20poly1305
/usr/share/doc/openssh-7.4p1/PROTOCOL.key
/usr/share/doc/openssh-7.4p1/PROTOCOL.krl
/usr/share/doc/openssh-7.4p1/PROTOCOL.mux
/usr/share/doc/openssh-7.4p1/README
[...]

Find what scripts run when the package is installed.

[[email protected]tartarus ~]# rpm -q --scripts openssh
preinstall scriptlet (using /bin/sh):
getent group ssh_keys >/dev/null || groupadd -r ssh_keys || :

Common Issues

Here is how you would fix a corrupted rpm database. This is probably the most common issue you will run into.

[[email protected] rpm]# rpm -qa
^Cerror: db5 error(11) from dbenv->open: Resource temporarily unavailable
error: cannot open Packages index using db5 - Resource temporarily unavailable (11)
error: cannot open Packages database in /var/lib/rpm
^C^C^Cerror: db5 error(11) from dbenv->open: Resource temporarily unavailable
error: cannot open Packages database in /var/lib/rpm
[[email protected] rpm]# rsync -av /var/lib/rpm rpmbackup
[[email protected] rpm]# rm -f /var/lib/rpm/__db.*
[[email protected] rpm]# rpm --rebuilddb
[[email protected] rpm]# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages
BDB5105 Verification of Packages succeeded.

Another scenario. You have tried the above steps, but instead of verification succeeding, Packages is still broken, and absolutely nothing else you know of is working. ‘Packages’ is the master package metadata file. You will need to take a backup, rebuild the rpmdb from scratch, recover, reimport, and then verify. Then CHECK if rpm/yum are OK. If not, see below.

[[email protected] ~]# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages
rpmdb_verify: BDB0521 Page 0: Incomplete metadata page
rpmdb_verify: /var/lib/rpm/Packages: BDB0090 DB_VERIFY_BAD: Database verification failed
BDB5105 Verification of /var/lib/rpm/Packages failed.
[[email protected] ~]# rsync -av /var/lib/rpm rpmbackup
sending incremental file list
rpm/
rpm/.dbenv.lock
rpm/.rpm.lock
rpm/Basenames
rpm/Conflictname
rpm/Dirnames
rpm/Group
rpm/Installtid
rpm/Name
rpm/Obsoletename
rpm/Packages
rpm/Providename
rpm/Requirename
rpm/Sha1header
rpm/Sigmd5
rpm/Triggername
 
sent 5,847,427 bytes  received 305 bytes  11,695,464.00 bytes/sec
total size is 5,844,998  speedup is 1.00
[[email protected] ~]# rm -fr /var/lib/rpm/*
[[email protected] ~]# rpm --initdb
[[email protected] ~]# mv /var/lib/rpm/Packages /var/lib/rpm/Packages.original
[[email protected] ~]# db_recover -h /var/lib/rpm
[[email protected] ~]# db_dump /var/lib/rpm/Packages.original | db_load Packages
[[email protected] ~]# /usr/lib/rpm/rpmdb_verify /var/lib/rpm/Packages
BDB5105 Verification of /var/lib/rpm/Packages succeeded.

If it is still broken (rpm -qa returns nothing / yum is broken) Only then proceed - Even then, you may want to consider rebuilding the server and file a bug report with rpm as the next steps may leave your server in an inconsistent state. Note this is for rhel 7.X, your mileage will vary on other releases. Additional note - I don’t know if this is an “accepted” solution. This simply worked for me. If this is rhel 4,5,6 - try this redhat solution first.

re-import all of the gpg-keys and reset the environment variables for yum.

[[email protected] rpmdb-indexes]# rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-*
[[email protected] rpmdb-indexes]# RELEASEVER=7
[[email protected] rpmdb-indexes]# ARCH=x86_64
[[email protected] rpmdb-indexes]# echo $RELEASEVER > /etc/yum/vars/releasever
[[email protected] rpmdb-indexes]# echo $ARCH > /etc/yum/vars/arch

If yum is still broken - try setting the following directive in the given file, note the releaseserver number would probably change depending on your platform.

file: /usr/lib/python2.X/site-packages/yum/config.py

startupconf.releasever = releaserver

OR

startupconf.releasever = 6

If yum works, then you’ll query the cached packages and attempt to reinstall them all. This will take a very long time.

[[email protected] rpmdb-indexes]# screen
[[email protected] rpmdb-indexes]# for i in $(find /var/lib/yum/yumdb/ -maxdepth 2 -type d| cut -d '-' -f 2- | grep -v yumdb |cut -d '.' -f1 | sed 's/-[0-9]$//g'| tr '\n' ' '); do yum install -y ${i}; done | tee reinstall.log

The previous will likely miss a few packages, check with the below.

[[email protected] rpmdb-indexes]# find /var/lib/yum/yumdb/ -maxdepth 2 -type d| cut -d '-' -f 2- | grep -v yumdb > packages.list
[[email protected] rpmdb-indexes]# for i in $(grep '^No ' reinstall.log | awk '{print $3}'); do grep $i packages.list ; done

At the end, the rpm database should be repopulated. Verify the packages as well. Please note however, there are some likely untracked packages. If they were installed outside of yum or weren’t in the cache from earlier, they’ll be on the system, but not in the rpm database.

[[email protected] rpmdb-indexes]# rpm -qa | wc -l
580
[[email protected] rpmdb-indexes]# rpm -Va

That concludes this article. If you’ve read this far, thank you!