OLS not responding

Discussion in 'OpenLiteSpeed Bug Reports' started by ckissi, Aug 21, 2017.

  1. ckissi

    ckissi Member

    Hi,

    Happened to me again. No response in front-end (domains on OLS) as well as OSL admin interface (ERR_TIMED_OUT)

    Code:
    root@buyvm4:~# ps aux | grep lsws
    root     22894  0.0  0.0   4512   836 ?        S    03:07   0:00 sh -c gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core* >> /tmp/m-xV0UnS
    root     22895  0.0  1.9  74828 20300 ?        S    03:07   0:00 gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core
    root     23189  0.0  0.1  16580  1956 pts/0    S+   03:25   0:00 grep --color=auto lsws
    Code:
    root@buyvm4:~# service lsws status
    * lsws.service - LSB: lshttpd
       Loaded: loaded (/etc/init.d/lsws; bad; vendor preset: enabled)
       Active: active (running) since Mon 2017-08-14 04:04:36 EDT; 6 days ago
         Docs: man:systemd-sysv-generator(8)
      Process: 581 ExecStart=/etc/init.d/lsws start (code=exited, status=0/SUCCESS)
       CGroup: /system.slice/lsws.service
               |-22887 lsphp
               |-22888 lsphp
               |-22894 sh -c gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core* >> /tmp/m-xV0UnS
               |-22895 gdb --batch --command=/usr/local/lsws//admin/misc/gdb-bt /usr/local/lsws//bin/lshttpd /tmp/lshttpd/core
               |-28143 openlitespeed (lshttpd - main
               `-28151 openlitespeed (lscgid
    
    Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
    
    Server Error log:
    Code:
    root@buyvm4:~# cat /usr/local/lsws/logs/error.log
    2017-08-15 17:05:43.074 [ERROR] [198.98.55.68:59136] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-15 20:23:53.661 [ERROR] [198.98.55.68:60884] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 97920, size: 0, next: 0
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 119376, size: 0, next: 97920
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 128640, size: 0, next: 119376
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 137856, size: 0, next: 128640
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 13248, size: 0, next: 137856
    2017-08-16 03:00:08.026 [ERROR] [198.98.55.68:36192] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 154320, size: 0, next: 13248
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 38848, size: 0, next: 154320
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 117680, size: 0, next: 38848
    2017-08-16 08:23:47.328 [ERROR] [198.98.55.68:39216] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-16 14:28:25.615 [ERROR] [198.98.55.68:42256] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 150112, size: 0, next: 117680
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 152144, size: 0, next: 150112
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 157280, size: 0, next: 152144
    2017-08-16 17:05:47.035 [ERROR] [198.98.55.68:43722] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-16 20:23:56.724 [ERROR] [198.98.55.68:45072] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-16 20:38:38.508 [ERROR] [198.98.55.68:45102] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 135168, size: 0, next: 157280
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 53184, size: 0, next: 135168
    2017-08-17 08:23:55.510 [ERROR] [198.98.55.68:47482] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 61024, size: 0, next: 53184
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 115696, size: 0, next: 61024
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 108128, size: 0, next: 115696
    2017-08-17 11:05:58.796 [ERROR] [198.98.55.68:48202] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 160320, size: 0, next: 108128
    2017-08-17 17:01:13.469 [ERROR] [198.98.55.68:49322] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-17 17:06:14.808 [ERROR] [198.98.55.68:49334] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 31344, size: 0, next: 160320
    2017-08-17 20:24:03.786 [ERROR] [198.98.55.68:50162] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 53872, size: 0, next: 31344
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 109536, size: 0, next: 53872
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 141936, size: 0, next: 109536
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 23168, size: 0, next: 141936
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 125568, size: 0, next: 23168
    2017-08-18 07:00:42.549 [ERROR] [198.98.55.68:54544] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-18 08:23:52.613 [ERROR] [198.98.55.68:55150] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 155248, size: 0, next: 125568
    2017-08-18 10:17:51.491 [ERROR] [198.98.55.68:56318] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 10880, size: 0, next: 155248
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 28672, size: 0, next: 10880
    2017-08-18 17:06:25.475 [ERROR] [198.98.55.68:59924] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 8192, size: 0, next: 28672
    2017-08-18 20:23:48.336 [ERROR] [198.98.55.68:33246] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-19 06:00:09.118 [ERROR] [198.98.55.68:37932] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-19 08:24:49.851 [ERROR] [198.98.55.68:39098] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 100320, size: 0, next: 8192
    2017-08-19 17:05:55.918 [ERROR] [198.98.55.68:43298] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 19072, size: 0, next: 100320
    2017-08-19 20:24:25.069 [ERROR] [198.98.55.68:44710] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-19 22:40:03.196 [ERROR] [198.98.55.68:45340] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-19 23:39:26.335 [ERROR] [198.98.55.68:45492] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-20 06:01:52.591 [ERROR] [198.98.55.68:46430] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-20 08:23:46.711 [ERROR] [198.98.55.68:46830] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    2017-08-20 17:05:49.626 [ERROR] [198.98.55.68:48386] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    [DEBUG] [SHM] [-1-26:0x19ec0c0] release to freebucket, offset: 66496, size: 0, next: 19072
    2017-08-20 20:23:56.481 [ERROR] [198.98.55.68:49218] [modcompress---] compressbuf in 0, return -1 (written 0, flag in 1)
    
    Code:
    root@buyvm4:~# df -h
    Filesystem      Size  Used Avail Use% Mounted on
    udev            479M     0  479M   0% /dev
    tmpfs           100M   13M   87M  13% /run
    /dev/vda1        19G  9.3G  8.5G  53% /
    tmpfs           497M  184K  496M   1% /dev/shm
    tmpfs           5.0M     0  5.0M   0% /run/lock
    tmpfs           497M     0  497M   0% /sys/fs/cgroup
    
    I have to restart the lsws service to make it work again.
     
  2. ckissi

    ckissi Member

    UPDATE:

    restart of lsws didn't resolve the issue. I had to restart the whole VPS.
     
  3. lsfoo

    lsfoo Administrator

    Hi @ckissi

    Did you get a core dump from when this occurred? It could be a SHM corruption. There should have been a core file generated based on the ps output.

    Cheers,
    Kevin
     
  4. ckissi

    ckissi Member

    Sorry Kevin, I don't get it.

    Can you describe step by step what should I do when it happens again? What commands with which arguments to run?
     
  5. lsfoo

    lsfoo Administrator

    Please check /tmp/lshttpd/ (maybe bak-core). There is likely a core file there. If the timestamp matches about the time OLS stopped working, it was probably generated by that issue.

    If this is the case, please open a private conversation with me and send me the output after the following commands:
    Code:
    > gdb /usr/local/lsws/bin/openlitespeed /tmp/lshttpd/bak_core/core
    > bt
     
  6. lsfoo

    lsfoo Administrator

    once outputted, you can type 'quit' to quit.
     
  7. ckissi

    ckissi Member

    This is the list of the folder currently:

    Code:
    root@buyvm4:/tmp/lshttpd# ll
    drwxr-xr-x  3 nobody nogroup   4096 Aug 21 03:48 ./
    drwxrwxrwt  9 root   root    208896 Aug 21 10:27 ../
    -rw-r--r--  1 root   root         5 Aug 21 03:47 lshttpd.pid
    srwxr-xr-x  1 nobody nogroup      0 Aug 21 03:48 lsphp56.sock=
    srwxr-xr-x  1 nobody nogroup      0 Aug 21 03:48 lsphp.sock=
    -rw-r--r--  1 nobody nogroup   1370 Aug 21 10:28 .rtreport
    -rw-r--r--  1 root   nogroup   1198 Aug 21 03:47 .status
    drwx------ 18 nobody nogroup   4096 Aug 21 03:48 swap/
    No dump there. So probably I have to wait when it'll happen again.
     
  8. lsfoo

    lsfoo Administrator

    Just to confirm, you have gdb installed there correct?

    In that case, yeah - it looks like it's trying to create a core file, but did not yet.
     
  9. ckissi

    ckissi Member

    Code:
    root@buyvm4:~# gdb --version
    GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
    Copyright (C) 2016 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
    and "show warranty" for details.
    This GDB was configured as "x86_64-linux-gnu".
    Type "show configuration" for configuration details.
    For bug reporting instructions, please see:
    <http://www.gnu.org/software/gdb/bugs/>.
    Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.
    For help, type "help".
    Type "apropos word" to search for commands related to "word".
     
  10. lsfoo

    lsfoo Administrator

    Well, until it creates a core file, unfortunately, we cannot do much to figure out what the issue is.

    Please try deleting the contents of /dev/shm/ols in the meantime and restart the server, hopefully the issue will present itself.
     
  11. ckissi

    ckissi Member

    What these files actually are ?
    Code:
    -rw------- 1 nobody nogroup   8192 Aug 21 03:38 adns_cache.lock
    -rw------- 1 nobody nogroup 131072 Aug 21 11:35 adns_cache.shm
    
     
  12. lsfoo

    lsfoo Administrator

    They cache information for later in case it is needed again.

    These specific ones are for dns.
     

Share This Page