Troubleshooting an application that wouldn’t come up. I found that apache refused to start…
In the log I found:
[emerg] (28)No space left on device: Couldn't create accept lock
Disk space was fine, and we are not using quotas on the application filesystem.
That leaves semaphores… I always have to go find the commands to look through these and remove/diagnose.
[root@server logs]# for semid in `ipcs -s | grep nobody | cut -f2 -d" "`; do ipcrm -s $semid; done [root@server logs]# ipcs -s ------ Semaphore Arrays -------- key semid owner perms nsems 0x000000a7 0 root 600 1 0x00000000 16809985 apache 600 1 0x00000000 16842754 apache 600 1 0x00000000 16941059 apache 600 1 0x00000000 16973828 apache 600 1 0x00000000 17072133 apache 600 1 0x00000000 17104902 apache 600 1 0x00000000 17203207 apache 600 1 0x00000000 17235976 apache 600 1 0x00000000 17465353 apache 600 1 (these go on for many lines...)
[root@server logs]#
[root@server logs]# for i in `ipcs -s | grep apache | awk '{ print $2 }'`
> do
> echo $i
> done
16809985
16842754
16941059
16973828
17072133
17104902
17203207
17235976
17465353
17498122
(etc)
That gives me the handle to remove, and then
[root@server logs]# for i in `ipcs -s | grep apache | awk '{ print $2 }'`; do echo $i; ipcrm -s $i; done
16809985
16842754
16941059
16973828
17072133
17104902
17203207
17235976
17465353
17498122
(etc)
removes them.
next is to verify they are gone, and to start apache
[root@server logs]# ipcs -s
------ Semaphore Arrays --------
key semid owner perms nsems
0x000000a7 0 root 600 1
[root@server logs]# for i in `ipcs -s | grep apache | awk '{ print $2 }'`; do echo $i; ipcrm -s $i; done
[root@server logs]# ipcs -s
[root@server logs]# /sbin/service httpd start
Starting httpd:
[root@server logs]# ps -ef | grep http
root 22571 1 0 11:00 ? 00:00:00 /usr/sbin/httpd.worker
root 22572 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app1-error_current /opt/cc/logs/http/app1-error.%Y-%m-%d-00_00_00.log
root 22573 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app1_current /opt/logs/http/app1-error.%Y-%m-%d-00_00_00.log
root 22574 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app2-error_current /opt/logs/http/app2-error.%Y-%m-%d-00_00_00.log
root 22575 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app2_current /opt/logs/http/app2-error.%Y-%m-%d-00_00_00.log
root 22578 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app1-access_current /opt/logs/http/app1-access.%Y-%m-%d-00_00_00.log
root 22579 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app1-access_current /opt/logs/http/app1-access.%Y-%m-%d-00_00_00.log
root 22580 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app2-access_current /opt/logs/http/app2-access.%Y-%m-%d-00_00_00.log
root 22581 22571 0 11:00 ? 00:00:00 /usr/sbin/cronolog -z EST -H /opt/logs/http/app2-access_current /opt/logs/http/app2-access.%Y-%m-%d-00_00_00.log
apache 22585 22571 0 11:00 ? 00:00:00 /usr/sbin/httpd.worker
apache 22586 22571 0 11:00 ? 00:00:00 /usr/sbin/httpd.worker
root 22688 17168 0 11:00 pts/0 00:00:00 grep http
Done.
–doug