Auditing systemd : How To Solving Failed Units with Systemctl Centos 7
Auditing systemd
Solving failed units with systemctl
Systemd is an alternative service manager to the more traditional init system. To ensure the system is healthy, failed units should be investigated on a regular basis. Sooner or later a unit might fail and showing up the systemctl listing. In this article we have a look at how to solve it.
 |
| How To Solve solving failed units with systemctl |
Command to show : systemctl --failed
Why do services fail?
During the start of the system, enabled services are started and queued to be executed. Most processes will start correctly and systemd logs the related status in the journal. However, in some cases a service might enter a “failed state”, as a result of another command not finishing properly.
[root@localhost ~]# systemctl systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION * auditd.service loaded failed failed Security Auditing Service * httpd.service loaded failed failed The Apache HTTP Server * kdump.service loaded failed failed Crash recovery kernel arming * plymouth-start.service loaded failed failed Show Plymouth Boot Screen * systemd-sysctl.service loaded failed failed Apply Kernel Variables * systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories * systemd-vconsole-setup.service loaded failed failed Setup Virtual Console * vzquota.service loaded failed failed LSB: Start vzquota at the end of boot LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type. 8 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'.
[root@localhost ~]$ systemctl list-units
UNIT LOAD ACTIVE SUB DESCRIPTION
sys-devices-virtual-net-gre0.device loaded active plugged /sys/devices/virtual/net/gre0
sys-devices-virtual-net-gretap0.device loaded active plugged /sys/devices/virtual/net/gretap0
sys-devices-virtual-net-venet0.device loaded active plugged /sys/devices/virtual/net/venet0
sys-subsystem-net-devices-gre0.device loaded active plugged /sys/subsystem/net/devices/gre0
sys-subsystem-net-devices-gretap0.device loaded active plugged /sys/subsystem/net/devices/gretap0
sys-subsystem-net-devices-venet0.device loaded active plugged /sys/subsystem/net/devices/venet0
-.mount loaded active mounted /
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
brandbot.path loaded active waiting Flexible branding
systemd-ask-password-console.path loaded active waiting Dispatch Password Requests to Console Directory Watch
systemd-ask-password-wall.path loaded active waiting Forward Password Requests to Wall Directory Watch
session-3861863.scope loaded active running Session 3861863 of user ninxia07
session-3863031.scope loaded active running Session 3863031 of user ninxia07
session-3864324.scope loaded active running Session 3864324 of user ninxia07
atd.service loaded active running Job spooling tools
* auditd.service loaded failed failed Security Auditing Service
console-getty.service loaded active running Console Getty
crond.service loaded active running Command Scheduler
dbus.service loaded active running D-Bus System Message Bus
firewalld.service loaded active running firewalld - dynamic firewall daemon
getty@tty2.service loaded active running Getty on tty2
* httpd.service loaded failed failed The Apache HTTP Server
* kdump.service loaded failed failed Crash recovery kernel arming
lvm2-lvmetad.service loaded active running LVM2 metadata daemon
lvm2-monitor.service loaded active exited Monitoring of LVM2 mirrors, snapshots etc. using dmeven
Services usually fail because of a missing dependency (e.g. a file or mount point), missing configuration, or incorrect permissions. In this example we see that the dev-mqueue unit with type mount fails. As the type is a mount, the reason is most likely because mounting a particular partition failed.
By using the systemctl status command we can see the details of the dev-mqueue.mount unit:
[root@localhost ~]# systemctl status auditd.service
* auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2017-04-06 11:30:50 CDT; 1h 14min ago
Docs: man:auditd(8)
https://people.redhat.com/sgrubb/audit/
Main PID: 91 (code=exited, status=4)
Mar 06 00:04:32 migandul systemd[1]: Failed to start Security Auditing Service.
Mar 06 00:04:32 migandul systemd[1]: Unit auditd.service entered failed state.
Mar 06 00:11:17 migandul systemd[1]: Starting Security Auditing Service...
Mar 06 00:11:17 migandul auditd[3420]: You must be root or have capabilities to run this program.
Mar 06 00:11:17 migandul systemd[1]: auditd.service: main process exited, code=exited, status=4/NOPERMISSION
Mar 06 00:11:17 migandul augenrules[3421]: /sbin/augenrules: No change
Mar 06 00:11:17 migandul augenrules[3421]: You must be root to run this program.
Mar 06 00:11:17 migandul systemd[1]: Failed to start Security Auditing Service.
Mar 06 00:11:17 migandul systemd[1]: Unit auditd.service entered failed state.
Mar 06 00:11:17 migandul systemd[1]: auditd.service failed.
This shows the related command which was executed. We see the unit failed on exit-code as it was not the expected value of 0 (actually it is 32). Manually running the command shows the device /dev/mqueue is missing.
Similar to this service, IPMI fails on our virtual machine. As there is no /dev/ipmi* device, the service can’t start and fails:
[root@localhost ~]# systemctl status auditd.service ? auditd.service - auditd Daemon * auditd.service - Security Auditing Service Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2017-04-06 11:30:50 CDT; 55min ago Docs: man:auditd(8) https://people.redhat.com/sgrubb/audit/ Process: 92 ExecStartPost=/sbin/augenrules --load (code=exited, status=4) Process: 91 ExecStart=/sbin/auditd -n (code=exited, status=4) Main PID: 91 (code=exited, status=4) Unit -.service could not be found. * auditd.service - Security Auditing Service Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2017-04-06 11:30:50 CDT; 55min ago Docs: man:auditd(8) https://people.redhat.com/sgrubb/audit/ Process: 92 ExecStartPost=/sbin/augenrules --load (code=exited, status=4) Process: 91 ExecStart=/sbin/auditd -n (code=exited, status=4) Main PID: 91 (code=exited, status=4) Unit -.service could not be found. * auditd.service - Security Auditing Service Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2017-04-06 11:30:50 CDT; 55min ago Docs: man:auditd(8) https://people.redhat.com/sgrubb/audit/ Process: 92 ExecStartPost=/sbin/augenrules --load (code=exited, status=4) Process: 91 ExecStart=/sbin/auditd -n (code=exited, status=4) Main PID: 91 (code=exited, status=4) Unit Daemon.service could not be found.
Clearing failed units
You can manually clear out failed units with the systemctl reset-failed command. This can be done for all units, or a single one.
Services which are no longer needed, are better to be stopped and disabled.
systemctl stop auditd.service
systemctl disable auditd.service
That’s all! Explained How To Solving Failed Units with Systemctl Centos 7.
Comments
Post a Comment