Insomnia247.NL was scheduled to be offline for 30 minutes on Saturday 6th July 2024 for maintenance so I did not log onto Insomnia247.NL at all on that Saturday. I logged onto Insomnia247.NL on Sunday morning 7th July 2024. A big change related to a quota on the filesystem for these live webserver logs was made, so I became immediately warned that files for myself on this filesystem were not being written. Cf. below. So I deleted files off this filesystem just before uptime hinted that these webservers were on for some 17 hours and 27 minutes without new entries saved into these logfiles. However, these webserver logs were actually written to during parts of these days but maybe different attempts to write to them after 2:14:23a.m. on Sunday 07/07/2024 failed. login as: gloucester Pre-authentication banner message from server: | Lydia7 on the Insomnia 24/7 network. | Lost your password? https://www.insomnia247.nl/lostpass.html | End of banner message from server Keyboard-interactive authentication prompts from server: | Password: End of keyboard-interactive prompts from server Welcome to ▀ ▄ ▀▄ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▄▄▄▄▄▄ ▄▄▄▄▄▄ ▄▄ ▀ ▀ ▀ ▄▄▄▄▄▄▄▄▄ ▄▄ ▄ ▀▀ ▄ ▄ ▀ ▄ ▀ ▀ ▀▀ ▀ ▀ ▀ ▀ ▀ ▄ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀▀▀ ▀ ▀ ▀ ▀ ▄ ▀ ▀ ▀ ▀ ▄▄ ▀ ▀ ▀ ▀ ▀ ▄ ▀ ▀ ▀ ▀ ▀ ▀ ▀▀▀▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▄▄▄ ▀ ▀ ▀ ▀ ▀▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀ ▀▀▀▀▀▀▀▀▀▀▀▀▀▀ ▄▄▄ ▄ ▀▀▀ ▀ ▀ ▀▀ ▀▀ ▀▀ ▀▀ ▀▀ ▀ ▀ Insomnia 24/7 ▀▀▀▀▀ ┏━━ CPU ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┃ Type 24 x AMD Ryzen 9 3900X @ 3.8GHz ┃ Load 1.42 % ┣━━ Memory ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┃ Free 37297M ┃ Total 64222M ┣━━ Disk (Free / Total) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┃ SSD array 983G / 1.8T ┃ Coldstorage 1.1T / 3.7T ┣━━ Users ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┃ Total 841 ┃ Active this month 41 ┃ Online now 10 ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (0.0s) ━━ ┏━━ News ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┃ All is well. Have a nice day! ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (0.0s) ━━ ┏━━ Message for gloucester ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ┃ Your mail was received. Logs don't go back as far as you asked for, but since then no entries matching your query have appeared. ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ (0.0s) ━━ md0: write failed, group block limit reached. cp: error copying '/home/gloucester/.bash_history' to '/home/gloucester/backed_up_shell_histories/bash_history_timestamped_2024-07-05_23:27_backed_up_by_Process586701_on_2024-07-07_10:54': Disk quota exceeded /home/gloucester $ quota -v gloucester Disk quotas for user gloucester (uid 1998): Filesystem blocks quota limit grace files quota limit grace /dev/md0 12597276 0 0 129489 0 0 /home/gloucester $ df -h Filesystem Size Used Avail Use% Mounted on udev 32G 0 32G 0% /dev tmpfs 6.3G 1.9M 6.3G 1% /run /dev/md0 1.8T 757G 983G 44% / tmpfs 32G 432K 32G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup /dev/sda1 93M 5.9M 87M 7% /boot/efi /dev/sdb1 3.7T 2.6T 1.1T 70% /coldstorage tmpfs 6.3G 80K 6.3G 1% /run/user/1000 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1914 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2581 tmpfs 6.3G 276K 6.3G 1% /run/user/2631 tmpfs 6.3G 128K 6.3G 1% /run/user/2587 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2647 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2457 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1496 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2459 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1990 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2197 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2645 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2677 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1970 tmpfs 6.3G 80K 6.3G 1% /run/user/2251 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2521 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2062 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2431 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1554 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2591 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1846 tmpfs 6.3G 4.0K 6.3G 1% /run/user/2613 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1136 tmpfs 6.3G 4.0K 6.3G 1% /run/user/1998 /home/gloucester $ info quota /home/gloucester $ info quota /home/gloucester $ quota --user gloucester Disk quotas for user gloucester (uid 1998): none /home/gloucester $ quota --group Disk quotas for group users (gid 100): none Disk quotas for group gloucester (gid 1501): Filesystem blocks quota limit grace files quota limit grace /dev/md0 10755756* 10240000 10496000 6days 129487 0 0 /home/gloucester $ quota --group --human-readable Disk quotas for group users (gid 100): none Disk quotas for group gloucester (gid 1501): Filesystem space quota limit grace files quota limit grace /dev/md0 10504M* 10000M 10250M 6days 130k 0 0 /home/gloucester $ quota --group --human-readable --filesystem=/home/gloucester/coldstorage Disk quotas for group users (gid 100): none Disk quotas for group gloucester (gid 1501): Filesystem space quota limit grace files quota limit grace /dev/md0 10504M* 10000M 10250M 6days 130k 0 0 [. . . /home/gloucester $ rm -R geant4_workdir] /home/gloucester $ quota --group --human-readable Disk quotas for group users (gid 100): none Disk quotas for group gloucester (gid 1501): Filesystem space quota limit grace files quota limit grace /dev/md0 10503M* 10000M 10250M 6days 130k 0 0 [. . . /home/gloucester $ rm -R profilers] [. . . /home/gloucester $ rm -R temp] /home/gloucester $ quota --group --human-readable Disk quotas for group users (gid 100): none Disk quotas for group gloucester (gid 1501): Filesystem space quota limit grace files quota limit grace /dev/md0 9712M 10000M 10250M 33552 0 0 /home/gloucester $ uptime 11:07:23 up 17:27, 19 users, load average: 0.86, 0.54, 0.43 [. . .] /home/gloucester $ cd http_logs /home/gloucester/http_logs $ ls -l -t -r total 51464 -rw-r--r-- 1 gloucester gloucester 17341 Jul 9 2021 mod_dir.html -rw-r--r-- 1 gloucester gloucester 57660 Jul 9 2021 mod_autoindex.html -rw-r--r-- 1 gloucester gloucester 37261 Jul 9 2021 logs.html -rw-r--r-- 1 gloucester gloucester 27063 Jul 9 2021 htaccess.html -rw-r--r-- 1 gloucester gloucester 37717 Jul 9 2021 directives.html -rw-r--r-- 1 gloucester gloucester 16477 Jul 9 2021 configuring.html -rw------- 1 gloucester gloucester 396 Apr 23 14:08 Webbrowser_IDs.txt~ -rw------- 1 gloucester gloucester 550 Apr 27 16:52 Webbrowser_IDs.txt -rw-r--r-- 1 gloucester gloucester 22602 Jun 4 15:58 Creative_Commons_Corporation_Attribution-ShareAlike_4.0_International_License.HTM -rw-r--r-- 1 gloucester gloucester 874 Jun 26 14:20 Dh_iarr_seo_ar_an_2024-06-25.txt -rwxr-xr-x 1 gloucester gloucester 2425 Jun 26 15:49 Na_h-amhairc_orra.sh.bak1 -rwxr-xr-x 1 gloucester gloucester 2429 Jun 26 15:58 Na_h-amhairc_orra.sh.bak2 -rw-r--r-- 1 gloucester gloucester 725 Jun 26 16:00 Dh_iarr_seo_ar_an_2024-06-24.txt -rwxr-xr-x 1 gloucester gloucester 4860 Jun 26 16:03 Na_h-amhairc_orra.sh.bak3 -rwxr-xr-x 1 gloucester gloucester 4856 Jun 26 16:04 Na_h-amhairc_orra.sh.bak4 -rw-r--r-- 1 gloucester gloucester 83 Jun 26 16:05 Dh_iarr_seo_ar_an_2024-06-23.txt -rw-r--r-- 1 gloucester gloucester 349 Jun 26 16:08 Dh_iarr_seo_ar_an_2024-06-22.txt -rw-r--r-- 1 gloucester gloucester 566 Jun 26 16:11 Dh_iarr_seo_ar_an_2024-06-21.txt -rw-r--r-- 1 gloucester gloucester 634 Jun 26 16:13 Dh_iarr_seo_ar_an_2024-06-20.txt -rw-r--r-- 1 gloucester gloucester 3047 Jun 26 16:15 Dh_iarr_seo_ar_na_2024-06-1x.txt -rwxr-xr-x 1 gloucester gloucester 18560 Jun 26 16:16 Na_h-amhairc_orra.sh~ -rwxr-xr-x 1 gloucester gloucester 18560 Jun 26 16:16 Na_h-amhairc_orra.sh.bak5 -r-xr-xr-x 1 gloucester gloucester 19031 Jun 26 16:25 Na_h-amhairc_orra.sh.bak6.ro-mhor -r-xr-xr-x 1 gloucester gloucester 19032 Jun 26 16:26 Na_h-amhairc_orra.sh.bak7.neo-ro-mhor -rwxr-xr-x 1 gloucester gloucester 19032 Jun 26 16:26 Na_h-amhairc_orra.sh -rw-r--r-- 1 gloucester gloucester 483019 Jun 27 15:50 geolite2-free-geolocation-data.HTM -rw-r--r-- 1 gloucester gloucester 51414290 Jun 28 04:00 GeoLite2-City_dated_2024-06-28_from_P3TERX.mmdb drwxr-x--- 3 gloucester gloucester 4096 Jun 30 13:18 . drwxr-xr-x 7 gloucester gloucester 4096 Jul 1 18:27 coimheas -rw-r--r-- 1 root root 1725 Jul 2 16:12 gloucester.insomnia247.nl_error.log -rw-r--r-- 1 root root 4302 Jul 3 14:30 gloucester.insomnia247.nl_ssl_error.log -rw-r--r-- 1 root root 80569 Jul 6 22:34 gloucester.insomnia247.nl_ssl_access.log -rw-r--r-- 1 root root 271442 Jul 7 02:14 gloucester.insomnia247.nl_access.log drwxr-x--- 31 gloucester gloucester 12288 Jul 7 11:06 .. /home/gloucester/http_logs $ fgrep '06/Jul/2024' *.log* gloucester.insomnia247.nl_access.log:52.230.152.82 - - [06/Jul/2024:06:50:24 +0200] "GET /robots.txt HTTP/1.1" 404 275 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.254 - - [06/Jul/2024:06:50:35 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.138 - - [06/Jul/2024:10:00:43 +0200] "GET /robots.txt HTTP/1.1" 404 275 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.76 - - [06/Jul/2024:10:00:53 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.91 - - [06/Jul/2024:20:20:19 +0200] "GET /robots.txt HTTP/1.1" 404 275 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.147 - - [06/Jul/2024:20:20:29 +0200] "GET /Englischunterricht/huhcorp HTTP/1.1" 301 347 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.168 - - [06/Jul/2024:21:19:54 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.36 - - [06/Jul/2024:21:47:27 +0200] "GET /robots.txt HTTP/1.1" 404 275 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.53 - - [06/Jul/2024:21:47:37 +0200] "GET /Englischunterricht/huhcorp HTTP/1.1" 301 347 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.236 - - [06/Jul/2024:22:32:14 +0200] "GET /Englischunterricht/huhcorp HTTP/1.1" 301 347 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.113 - - [06/Jul/2024:22:44:23 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.199 - - [06/Jul/2024:23:29:19 +0200] "GET /robots.txt HTTP/1.1" 404 275 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.0 - - [06/Jul/2024:23:29:21 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_ssl_access.log:34.219.188.32 - - [06/Jul/2024:10:59:45 +0200] "GET /Scholastica_or_LinkedIn_connivance/2023-12-22b_fee-waiver_request.pdf HTTP/1.1" 200 2049333 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.157 Safari/537.36" gloucester.insomnia247.nl_ssl_access.log:35.93.81.205 - - [06/Jul/2024:12:47:27 +0200] "GET /Scholastica_or_LinkedIn_connivance/2024-01-17a_disappeared_again_in_under_19_hours.pdf HTTP/1.1" 200 223349 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" gloucester.insomnia247.nl_ssl_access.log:66.249.72.163 - - [06/Jul/2024:22:34:53 +0200] "GET /robots.txt HTTP/1.1" 404 276 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" gloucester.insomnia247.nl_ssl_access.log:66.249.72.161 - - [06/Jul/2024:22:34:55 +0200] "GET /Scholastica_or_LinkedIn_connivance/2024-01-17a_disappeared_again_in_under_19_hours.pdf HTTP/1.1" 304 - "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.175 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" /home/gloucester/http_logs $ fgrep '07/Jul/2024' *.log* gloucester.insomnia247.nl_access.log:52.230.152.50 - - [07/Jul/2024:00:48:23 +0200] "GET /robots.txt HTTP/1.1" 404 275 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.52 - - [07/Jul/2024:00:48:24 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.144 - - [07/Jul/2024:01:24:19 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.128 - - [07/Jul/2024:01:52:23 +0200] "GET /robots.txt HTTP/1.1" 404 275 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.27 - - [07/Jul/2024:01:52:33 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" gloucester.insomnia247.nl_access.log:52.230.152.102 - - [07/Jul/2024:02:14:23 +0200] "GET /Englischunterricht/huhcorp/ HTTP/1.1" 200 535 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" /home/gloucester/http_logs $ ls -l -t -r ~/coldstorage/http_logs total 952 drwxr-xr-x 2 gloucester gloucester 4096 Jul 8 2023 2019-07-24_to_2023-07-01 dr-xr-xr-x 2 gloucester gloucester 4096 Oct 6 2023 2023-08-11_to_2023-10-06 drwxr-xr-x 2 gloucester gloucester 4096 Nov 4 2023 2023-10-01_to_2023-11-04 drwxr-xr-x 2 gloucester gloucester 58 Nov 13 2023 2023-11-04_to_2023-11-13 drwxr-xr-x 2 gloucester gloucester 4096 Feb 2 22:58 2023-11-13_to_2024-02-02 drwxr-xr-x 2 gloucester gloucester 110 Feb 11 18:49 2024-02-02_to_2024-02-09 drwxr-xr-x 2 gloucester gloucester 4096 Mar 18 21:30 2024-02-09_to_2024-03-18 -rw-r--r-- 1 root root 267 Apr 20 00:00 gloucester.insomnia247.nl_ssl_error.log.3.gz drwxr-xr-x 2 gloucester gloucester 59 Apr 24 11:46 unique_webserver_log_on_2024-04-24_file_is_timestamped_2019-10-13_entry_is_timestamped_2019-10-12 -rw-r--r-- 1 root root 2213 May 1 00:00 gloucester.insomnia247.nl_access.log.3.gz -rw-r--r-- 1 root root 3075 May 1 00:00 gloucester.insomnia247.nl_ssl_access.log.3.gz drwxr-xr-x 2 gloucester gloucester 4096 May 5 10:19 2024-03-18_to_2024-05-05 -rw-r--r-- 1 root root 326 May 21 00:00 gloucester.insomnia247.nl_error.log.3.gz -rw-r--r-- 1 root root 4641 Jun 1 00:01 gloucester.insomnia247.nl_access.log.2.gz -rw-r--r-- 1 root root 7554 Jun 1 00:01 gloucester.insomnia247.nl_ssl_access.log.2.gz -rw-r--r-- 1 root root 273 Jun 2 06:25 gloucester.insomnia247.nl_ssl_error.log.2.gz -rw-r--r-- 1 root root 256 Jun 16 00:00 gloucester.insomnia247.nl_error.log.2.gz drwxr-xr-x 2 gloucester gloucester 4096 Jun 30 18:38 2024-05-01_to_2024-06-30 -rw-r--r-- 1 root root 293827 Jul 1 00:01 gloucester.insomnia247.nl_access.log.1 -rw-r--r-- 1 root root 629 Jul 1 00:01 gloucester.insomnia247.nl_error.log.1 -rw-r--r-- 1 root root 576456 Jul 1 00:01 gloucester.insomnia247.nl_ssl_access.log.1 drwxr-x--- 93 gloucester gloucester 16384 Jul 1 12:55 .. -rw-r--r-- 1 root root 1657 Jul 2 06:25 gloucester.insomnia247.nl_ssl_error.log.1 drwxr-x--- 12 gloucester gloucester 4096 Jul 2 06:25 . /home/gloucester/http_logs $