1 answer
- 10-1
Yes, it most definitely should. What are you not seeing that you're expecting to see? Have you checked the discovery log to make sure you're getting a full audit (credentials are working, the audit script is working)?
- Jason
Yes, for an example system (IP 10.85.8.32 ), I see it was scanned again, however not clear that it ran a new audit against it. Below is the log. I've hidden sensitive items with "HIDDEN"
For the system itself, the "Last seen on" date is 2020-07-03 17:42:31 and the software inventory is not up to date.
Would one of these configuration options prevent re-inventorying a system? Some other configuration setting?
match_dbus n system 2000-01-01 00:00:00 Should we match a device based on its dbus id. match_fqdn y system 2000-01-01 00:00:00 Should we match a device based on its fqdn. match_dns_fqdn n system 2000-01-01 00:00:00 Should we match a device based on its DNS fqdn. match_dns_hostname n system 2000-01-01 00:00:00 Should we match a device based on its DNS hostname. match_hostname y system 2000-01-01 00:00:00 Should we match a device based only on its hostname. match_hostname_dbus y system 2000-01-01 00:00:00 Should we match a device based on its hostname and dbus id. match_hostname_serial y system 2000-01-01 00:00:00 Should we match a device based on its hostname and serial. match_hostname_uuid y system 2000-01-01 00:00:00 Should we match a device based on its hostname and UUID. match_ip y Administrator 2020-09-17 12:47:10 Should we match a device based on its ip. match_mac y Administrator 2020-01-11 21:39:55 Should we match a device based on its mac address. match_mac_vmware n system 2000-01-01 00:00:00 Should we match a device based mac address even if its a known likely duplicate from VMware. match_serial y system 2000-01-01 00:00:00 Should we match a device based on its serial number. match_serial_type y system 2000-01-01 00:00:00 Should we match a device based on its serial and type. match_sysname y system 2000-01-01 00:00:00 Should we match a device based only on its SNMP sysName. match_sysname_serial y system 2000-01-01 00:00:00 Should we match a device based only on its SNMP sysName and serial. match_uuid y system 2000-01-01 00:00:00 Should we match a device based on its UUID. match_dbus n system 2000-01-01 00:00:00 Should we match a device based on its dbus id. match_fqdn y system 2000-01-01 00:00:00 Should we match a device based on its fqdn. match_dns_fqdn n system 2000-01-01 00:00:00 Should we match a device based on its DNS fqdn. match_dns_hostname n system 2000-01-01 00:00:00 Should we match a device based on its DNS hostname. match_hostname y system 2000-01-01 00:00:00 Should we match a device based only on its hostname. match_hostname_dbus y system 2000-01-01 00:00:00 Should we match a device based on its hostname and dbus id. match_hostname_serial y system 2000-01-01 00:00:00 Should we match a device based on its hostname and serial. match_hostname_uuid y system 2000-01-01 00:00:00 Should we match a device based on its hostname and UUID. match_ip y Administrator 2020-09-17 12:47:10 Should we match a device based on its ip. match_mac y Administrator 2020-01-11 21:39:55 Should we match a device based on its mac address. match_mac_vmware n system 2000-01-01 00:00:00 Should we match a device based mac address even if its a known likely duplicate from VMware. match_serial y system 2000-01-01 00:00:00 Should we match a device based on its serial number. match_serial_type y system 2000-01-01 00:00:00 Should we match a device based on its serial and type. match_sysname y system 2000-01-01 00:00:00 Should we match a device based only on its SNMP sysName. match_sysname_serial y system 2000-01-01 00:00:00 Should we match a device based only on its SNMP sysName and serial. match_uuid y system 2000-01-01 00:00:00 Should we match a device based on its UUID.
53650145 2020-09-17 16:53:28 discover_subnet.sh logs Scanning Host: 10.85.8.32 notice 53650146 2020-09-17 16:53:29 discover_subnet.sh logs Nmap Command (Custom TCP Ports) 1.000000 notice /usr/bin/nmap -n -T4 -sS -p 22,135,62078 10.85.8.32 # Custom TCP Ports 53650147 2020-09-17 16:53:30 discover_subnet.sh logs Host 10.85.8.32 is up, received wmi (TCP port 135 open) response notice 135/tcp open msrpc /usr/bin/nmap -n -T4 -sS -p 22,135,62078 10.85.8.32 # Custom TCP Ports 53650148 2020-09-17 16:53:30 discover_subnet.sh logs Host 10.85.8.32 is up, received Nmap ping response notice Nmap done: 1 IP address (1 host up) scanned in 0.50 seconds /usr/bin/nmap -n -T4 -sS -p 22,135,62078 10.85.8.32 # Custom TCP Ports 53650149 2020-09-17 16:53:31 discover_subnet.sh logs Nmap Command (Custom UDP Ports) 1.000000 notice /usr/bin/nmap -n -T4 -sU -p 161 10.85.8.32 # Custom UDP Ports 53650150 2020-09-17 16:53:32 discover_subnet.sh logs Host 10.85.8.32 is up, received Nmap ping response notice Nmap done: 1 IP address (1 host up) scanned in 0.52 seconds /usr/bin/nmap -n -T4 -sU -p 161 10.85.8.32 # Custom UDP Ports 53650151 2020-09-17 16:53:33 discover_subnet.sh logs IP 10.85.8.32 responding, received open TCP port (135/tcp, msrpc), adding to device list. SSH Status: false, WMI Status: true, SNMP Status: false. 4.000000 (23 of 33) https://127.0.0.1/open-audit/index.php/input/discoveries 53650152 2020-09-17 16:53:33 m_device match WMI Status is true on 10.85.8.6 notice 53650153 2020-09-17 16:53:33 m_device match SSH Status is false on 10.85.8.6 notice 53650154 2020-09-17 16:53:33 m_device match SNMP Status is false on 10.85.8.6 notice 53650155 2020-09-17 16:53:33 discovery_helper discoveries Testing Windows credentials for 10.85.8.6 notice 53650156 2020-09-17 16:53:33 wmi_helper windows_credentials Windows credentials starting notice 53650157 2020-09-17 16:53:33 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.899158 success ["UUID","22903842-8194-DFE4-B389-AC959CDF0A57",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN****** --uninstall //10.85.8.6 "wmic csproduct get uuid" 2>&1 53650158 2020-09-17 16:53:34 input discoveries The discovery_id was used to successfully retrieve information for the discovery entry named VLAN 38 success 53650159 2020-09-17 16:53:34 input discoveries Received data for 10.85.8.32, now starting to process success 53650160 2020-09-17 16:53:34 input discoveries Set DNS FQDN for 10.85.8.32 in network_helper::dns_validate notice HIDDEN 53650161 2020-09-17 16:53:34 input discoveries Set DNS hostname for 10.85.8.32 in network_helper::dns_validate notice HIDDEN 53650162 2020-09-17 16:53:34 input discoveries Set DNS domain for 10.85.8.32 in network_helper::dns_validate notice HIDDEN 53650163 2020-09-17 16:53:34 m_device match Running devices::match function. notice 53650164 2020-09-17 16:53:34 m_device match Not running match_hostname_uuid, uuid not set. notice 53650165 2020-09-17 16:53:34 m_device match Not running match_hostname_dbus, dbus_identifier not set. notice 53650166 2020-09-17 16:53:34 m_device match Not running match_hostname_serial, serial not set. notice 53650167 2020-09-17 16:53:34 m_device match Not running match_dbus, matching rule set to: n. notice 53650168 2020-09-17 16:53:34 m_device match Not running match_dns_fqdn, matching rule set to: n. notice 53650169 2020-09-17 16:53:34 m_device match Not running match_dns_hostname, matching rule set to: n. notice 53650170 2020-09-17 16:53:34 m_device match Not running match_fqdn, fqdn not set. notice 53650171 2020-09-17 16:53:34 m_device match Not running match_serial_type, serial not set. notice 53650172 2020-09-17 16:53:34 m_device match Not running match_serial, serial not set. notice 53650173 2020-09-17 16:53:34 m_device match Not running match_sysname_serial, sysname not set. notice 53650174 2020-09-17 16:53:34 m_device match Not running match_sysname, sysname not set. notice 53650175 2020-09-17 16:53:34 m_device match Not running match_mac (ip table), mac_address not set. notice 53650176 2020-09-17 16:53:34 m_device match Not running match_mac (network table), mac_address not set. notice 53650177 2020-09-17 16:53:34 m_device match Not running match_mac (addresses), mac_addresses not set. notice 53650178 2020-09-17 16:53:34 m_device match HIT on IP Address (network table). success IP: 10.85.8.32, SystemID : 1259 53650179 2020-09-17 16:53:34 m_device match Device named HIDDEN found on initial Nmap result. success 53650180 2020-09-17 16:53:34 m_device match Delete the previous log entries for this device 0.002614 success /* input::discoveries */ DELETE FROM discovery_log WHERE system_id = 1259 and discovery_id != 9847 53650181 2020-09-17 16:53:34 m_device match Update the current log entries with our new device success /* input::discoveries */ UPDATE discovery_log SET system_id = 1259 WHERE discovery_id = 9847 and ip = '10.85.8.32'
- Jason
If I run a discovery of system directly (rather than through the subnet discovery), I get this:
53650640 2020-09-18 10:33:39 discover_subnet.sh logs Starting discovery for 10.85.8.32 start 53650641 2020-09-18 10:33:39 discover_subnet.sh logs Discovery for 10.85.8.32 using Nmap version 6.40 at /usr/bin/nmap notice 53650642 2020-09-18 10:33:40 discover_subnet.sh logs IPs in subnet: 1 notice nmap -n -sL 10.85.8.32 2>/dev/null | grep "Nmap done" | cut -d" " -f3 53650643 2020-09-18 10:33:40 discover_subnet.sh logs IPs after exclusions in subnet: 1 notice nmap -n -sL 10.85.8.32 2>/dev/null | grep "Nmap done" | cut -d" " -f3 53650644 2020-09-18 10:33:41 discover_subnet.sh logs IPs responding to Nmap ping in subnet (to be scanned): 1 notice nmap -n -oG - -sP 10.85.8.32 2>/dev/null | grep "Host:" | cut -d" " -f2 53650645 2020-09-18 10:33:42 discover_subnet.sh logs Updating discovery log with non-responding IPs notice 53650646 2020-09-18 10:33:42 discover_subnet.sh logs Scanning Host: 10.85.8.32 notice 53650647 2020-09-18 10:33:43 discover_subnet.sh logs Nmap Command (Custom TCP Ports) 1.000000 notice /usr/bin/nmap -n -T4 -sS -p 22,135,62078 10.85.8.32 # Custom TCP Ports 53650648 2020-09-18 10:33:44 discover_subnet.sh logs Host 10.85.8.32 is up, received wmi (TCP port 135 open) response notice 135/tcp open msrpc /usr/bin/nmap -n -T4 -sS -p 22,135,62078 10.85.8.32 # Custom TCP Ports 53650649 2020-09-18 10:33:45 discover_subnet.sh logs Host 10.85.8.32 is up, received Nmap ping response notice Nmap done: 1 IP address (1 host up) scanned in 0.55 seconds /usr/bin/nmap -n -T4 -sS -p 22,135,62078 10.85.8.32 # Custom TCP Ports 53650650 2020-09-18 10:33:46 discover_subnet.sh logs Nmap Command (Custom UDP Ports) notice /usr/bin/nmap -n -T4 -sU -p 161 10.85.8.32 # Custom UDP Ports 53650651 2020-09-18 10:33:46 discover_subnet.sh logs Host 10.85.8.32 is up, received Nmap ping response notice Nmap done: 1 IP address (1 host up) scanned in 0.52 seconds /usr/bin/nmap -n -T4 -sU -p 161 10.85.8.32 # Custom UDP Ports 53650652 2020-09-18 10:33:47 discover_subnet.sh logs IP 10.85.8.32 responding, received open TCP port (135/tcp, msrpc), adding to device list. SSH Status: false, WMI Status: true, SNMP Status: false. 4.000000 (1 of 1) https://127.0.0.1/open-audit/index.php/input/discoveries 53650653 2020-09-18 10:33:48 input discoveries The discovery_id was used to successfully retrieve information for the discovery entry named Device Discovery - HIDDEN success 53650654 2020-09-18 10:33:48 input discoveries Received data for 10.85.8.32, now starting to process success 53650655 2020-09-18 10:33:48 input discoveries Set DNS FQDN for 10.85.8.32 in network_helper::dns_validate notice HIDDEN.HIDDEN 53650656 2020-09-18 10:33:48 input discoveries Set DNS hostname for 10.85.8.32 in network_helper::dns_validate notice HIDDEN 53650657 2020-09-18 10:33:48 input discoveries Set DNS domain for 10.85.8.32 in network_helper::dns_validate notice HIDDEN 53650658 2020-09-18 10:33:48 m_device match Running devices::match function. notice 53650659 2020-09-18 10:33:48 m_device match Not running match_hostname_uuid, uuid not set. notice 53650660 2020-09-18 10:33:48 m_device match Not running match_hostname_dbus, dbus_identifier not set. notice 53650661 2020-09-18 10:33:48 m_device match Not running match_hostname_serial, serial not set. notice 53650662 2020-09-18 10:33:48 m_device match Not running match_dbus, matching rule set to: n. notice 53650663 2020-09-18 10:33:48 m_device match Not running match_dns_fqdn, matching rule set to: n. notice 53650664 2020-09-18 10:33:48 m_device match Not running match_dns_hostname, matching rule set to: n. notice 53650665 2020-09-18 10:33:48 m_device match Not running match_fqdn, fqdn not set. notice 53650666 2020-09-18 10:33:48 m_device match Not running match_serial_type, serial not set. notice 53650667 2020-09-18 10:33:48 m_device match Not running match_serial, serial not set. notice 53650668 2020-09-18 10:33:48 m_device match Not running match_sysname_serial, sysname not set. notice 53650669 2020-09-18 10:33:48 m_device match Not running match_sysname, sysname not set. notice 53650670 2020-09-18 10:33:48 m_device match Not running match_mac (ip table), mac_address not set. notice 53650671 2020-09-18 10:33:48 m_device match Not running match_mac (network table), mac_address not set. notice 53650672 2020-09-18 10:33:48 m_device match Not running match_mac (addresses), mac_addresses not set. notice 53650673 2020-09-18 10:33:48 m_device match HIT on IP Address (network table). success IP: 10.85.8.32, SystemID : 1259 53650674 2020-09-18 10:33:48 m_device match Device named HIDDEN found on initial Nmap result. success 53650675 2020-09-18 10:33:48 m_device match Delete the previous log entries for this device 0.010361 success /* input::discoveries */ DELETE FROM discovery_log WHERE system_id = 1259 and discovery_id != 9943 53650676 2020-09-18 10:33:48 m_device match Update the current log entries with our new device 50.006006 success /* input::discoveries */ UPDATE discovery_log SET system_id = 1259 WHERE discovery_id = 9943 and ip = '10.85.8.32' 53650677 2020-09-18 10:33:49 input discoveries Set discovery entry status to complete 53650678 2020-09-18 10:33:49 discover_subnet.sh logs Completed discovery, scanned 1 IP addresses 11.000000 finish 53650679 2020-09-18 10:34:38 m_device match WMI Status is true on 10.85.8.32 notice 53650680 2020-09-18 10:34:38 m_device match SSH Status is false on 10.85.8.32 notice 53650681 2020-09-18 10:34:38 m_device match SNMP Status is false on 10.85.8.32 notice 53650682 2020-09-18 10:34:38 discovery_helper discoveries Testing Windows credentials for 10.85.8.32 notice 53650683 2020-09-18 10:34:38 wmi_helper windows_credentials Windows credentials starting notice 53650684 2020-09-18 10:34:38 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.450474 success ["UUID","24BF3842-D0E4-6401-A3EB-17E68DAE3411",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic csproduct get uuid" 2>&1 53650685 2020-09-18 10:34:38 wmi_helper wmi_command Windows credentials complete. Credential set windows administrator working on 10.85.8.32 success 53650686 2020-09-18 10:34:38 wmi_helper wmi_audit WMI audit starting notice 53650687 2020-09-18 10:34:38 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.476761 success ["UUID","24BF3842-D0E4-6401-A3EB-17E68DAE3411",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic csproduct get uuid" 2>&1 53650688 2020-09-18 10:34:39 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.461463 success ["IdentifyingNumber","VMware-42 38 bf 24 e4 d0 01 64-a3 eb 17 e6 8d ae 34 11",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic csproduct get IdentifyingNumber" 2>&1 53650689 2020-09-18 10:34:39 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.467804 success ["Vendor","VMware, Inc.",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic csproduct get vendor" 2>&1 53650690 2020-09-18 10:34:40 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.477938 success ["Description","HIDDEN",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic os get description" 2>&1 53650691 2020-09-18 10:34:40 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.474132 success ["Name","HIDDEN",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic computersystem get name" 2>&1 53650692 2020-09-18 10:34:41 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.493441 success ["Domain","HIDDEN",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic computersystem get domain" 2>&1 53650693 2020-09-18 10:34:41 wmi_helper wmi_command Attempting to execute command using winexe-static-2 0.490732 success ["Name","Microsoft Windows Server 2012 R2 Datacenter|C:\\Windows|\\Device\\Harddisk0\\Partition2",""] timeout 1m /usr/local/open-audit/other/winexe-static-2 -U 'HIDDEN'%****** --uninstall //10.85.8.32 "wmic os get name" 2>&1 53650694 2020-09-18 10:34:42 wmi_helper wmi_audit WMI audit complete notice 53650695 2020-09-18 10:34:42 m_rules execute Running rules::match function. notice {"id":"1259","name":"HIDDEN","type":"computer","os_family":"","os_group":"Windows","sysDescr":"","last_seen":"2020-09-18 10:33:47","timestamp":"2020-09-18 10:33:47","ip":"10.85.8.32","audits_ip":"127.0.0.1","last_seen_by":"windows","discovery_id":"9943","credentials":"[3]","dns_hostname":"HIDDEN","dns_fqdn":"HIDDEN.HIDDEN","dns_domain":"HIDDEN","fqdn":"HIDDEN.HIDDEN","uuid":"24BF3842-D0E4-6401-A3EB-17E68DAE3411","serial":"VMware-42 38 bf 24 e4 d0 01 64-a3 eb 17 e6 8d ae 34 11","manufacturer":"VMware, Inc.","description":"HIDDEN","hostname":"HIDDEN","domain":"HIDDEN","os_name":"Microsoft Windows Server 2012 R2 Datacenter","install_dir":"C:\\Windows","where":"supplied"} Device Input (return). 53650696 2020-09-18 10:34:42 m_rules execute Hit on manufacturer VMware, Inc. li vmware 0.014236 notice {"form_factor":"Virtual"} Rules Match - Form Factor based on Manufacturer (like VMware), ID: 43 53650697 2020-09-18 10:34:42 m_rules execute Hit on form_factor Virtual eq Virtual Hit on os_group Windows eq Windows Hit on os_name Microsoft Windows Server 2012 R2 Datacenter li Server Hit on os_group is not empty Hit on class is empty 0.016354 notice {"class":"virtual server"} Rules Match - Class based on Form Factor and OS (Virtual Windows Server), ID: 49 53650698 2020-09-18 10:34:42 m_rules execute Completed rules::match function. 0.016354 notice 53650699 2020-09-18 10:34:42 include_input_discoveries discoveries Start of WINDOWS update for 10.85.8.32 notice 53650700 2020-09-18 10:34:42 audit_helper audit_convert Formatting system details notice 53650701 2020-09-18 10:34:42 audit_helper audit_convert Windows VMware style serial detected, creating vm_uuid. notice VMware-42 38 bf 24 e4 d0 01 64-a3 eb 17 e6 8d ae 34 11 -> 4238bf24-e4d0-0164-a3eb-17e68dae3411 53650702 2020-09-18 10:34:42 discovery_helper discoveries End of WINDOWS update for 10.85.8.32 notice 53650703 2020-09-18 10:34:42 include_input_discoveries discoveries Processing found ip addresses (non-snmp) for 10.85.8.32 notice 53650704 2020-09-18 10:34:42 include_input_discoveries discoveries Updating ip with ID 4192 notice 53650705 2020-09-18 10:34:42 include_input_discoveries discoveries Processing Nmap ports for 10.85.8.32 notice 53650706 2020-09-18 10:34:42 m_rules execute Running rules::match function. notice Device ID supplied: 1259 Device ID Input (update). 53650707 2020-09-18 10:34:42 m_rules execute Hit on manufacturer VMware, Inc. li vmware 0.010172 notice {"form_factor":"Virtual"} Rules Match - Form Factor based on Manufacturer (like VMware), ID: 43 53650708 2020-09-18 10:34:42 m_rules execute Completed rules::match function. 0.010172 notice 53650709 2020-09-18 10:34:42 include_input_discoveries discoveries At IP 10.85.8.32, discovery found a device of type 'computer'. 54.292378 notice 53650710 2020-09-18 10:34:42 include_input_discoveries discoveries Script details retrieved 54.292378 success /* discovery::process_subnet */ SELECT * FROM `scripts` WHERE `name` = 'audit_windows.vbs' AND `based_on` = 'audit_windows.vbs' ORDER BY `id` LIMIT 1 53650711 2020-09-18 10:34:42 include_input_discoveries discoveries Starting windows script audit for 10.85.8.32 notice 53650712 2020-09-18 10:34:43 wmi_helper copy_to_windows Linux attempt (SMB2 user@domain) to copy file to 10.85.8.32 succeeded in wmi_helper::copy_to_windows success smbclient -m SMB2 \\\\10.85.8.32\\\admin$ -U "HIDDEN*******" -c "put /usr/local/open-audit/other/scripts/audit_windows_20_09_18_10_34_42_52328300.vbs audit_windows.vbs 2>&1" 53650713 2020-09-18 10:34:43 wmi_helper execute_windows Using winexe-static-2 to run audit. success timeout 5m /usr/local/open-audit/other/winexe-static-2 -U "HIDDEN%******" --uninstall //10.85.8.32 "cscript C:\Windows\audit_windows.vbs submit_online=n create_file=w debugging=0 self_delete=y last_seen_by=audit_wmi system_id=1259 discov - Mark Unwin
Would one of these configuration options prevent re-inventorying a system? Some other configuration setting?
In short, no.
- Mark Unwin
What version of Open-AudIT are you using?
Are you using the GUI to run the discovery, or using the script discover_subnet.sh ?
You should only use the GUI. The script has been deprecated.
- Jason
I'm using 3.2.2
I tried the newer version of the newer discovery process again recently, however it hangs on discoveries sometimes and they end up in failed status or just run forever.
Yes, running it through the GUI only results in the above.
- Mark Unwin
I'd suggest using 3.4.0 as there have been multiple fixes since 3.2.2 and discover_subnet is no longer used. See:
Release Notes for Open-AudIT v3.3.0
Release Notes for Open-AudIT v3.3.1
- Jason
I upgraded to the latest version 3.4. Discovery mostly works on smaller subnets, but occasionally hangs (no updates in the discovery log for several hours) so I have to execute it again to get it to run. But at least existing systems are getting their details updated.
However now, on a larger subnet (a /19) it has hung and no discoveries will do anything now. cpu use is totally flat. I have 6,331 items in the queue table in my database. Seems I am back at the last place I was last time I tried this newer version (3.3) where discoveries just stop working. Here is my older question:
New discovery process in 3.3 does not work after upgrade from 3.2.2
- Mark Unwin
Jason,
If you call the below URL from the server, it should start the queue.
curl http://localhost/open-audit/index.php/util/queue
Once done, does the queue appear to be processing?
- Jason
So, eventually after ~24 hours, it finally marked the discovery as "failed" so then some other discoveries started actually trying to run, however now they all appear stuck again, no cpu, nothing.
I've tried curling that link, but no change in behavior.
Is there any way to see what is going on? This new discovery process seems odd in that it is idle quite often...
- Mark Unwin
In Enterprise (or Professional) on the discoveries list page, at the top right you'll see a drop down arrow. Click it and a panel should expand to show you the number of items in the queue, etc. There are some options to delete queue items, etc.
- Jason
Anything I can look at in community edition? The system appears totally idle and there is not much for logs.
Thanks
- Mark Unwin
You can try this on the command line.
mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM queue;" > queue_output.txt
That will dump the contents of the queue database table.
Now select a discovery that is not completing. Get its ID and run the below (substitue the actual ID for $id).
mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM discoveries WHERE id = $id;" > discovery_output.txt
And it's logs.
mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM discovery_log WHERE discovery_id = $id;" > discovery_log_output.txt
Parse the discovery log and see if there are any IP addresses that haven't been scanned that are in the queue_output.
Go to the end of the discovery_log and see if there is something amiss.
The last entry should resemble below (note, the output format will differ, but the contents should be the same). Note the message column and the command_status.
id: 10914 discovery_id: 7 system_id: NULL timestamp: 2020-09-24 15:37:04 severity: 7 severity_text: debug pid: 13952 ip: 127.0.0.1 file: wmi_helper function: copy_from_windows message: Discovery has finished. command: command_status: finished command_time_to_execute: 41.165920 command_output:If all else fails, email me those files at marku@opmantek.com and I can take a quick look to see if anything obvious stands out.
Mark.
- Jason
Thanks.
Below is info for one that is hung.
nothing is in the queue.
here is the discovery info:
id name org_id description type devices_assigned_to_org devices_assigned_to_location network_address system_id other options discard last_run last_finished duration status ip_all_count ip_responding_count ip_scanned_count ip_discovered_count ip_audited_count edited_by edited_date 9863 VLAN 96 1 Subnet - 10.84.96.0/19 subnet 1 1 http://127.0.0.1/open-audit/ 0 {"nmap":{},"match":{},"subnet":"10.84.96.0\\/19"} n 2020-09-28 20:15:12 2020-09-28 20:15:12 11:26:00 running 8192 6591 6591 3921 127 Administrator 2020-03-10 13:20:45Here are the last few log lines. Note the current time is 14:44
60536344 9863 128756 2020-09-29 06:22:22 7 debug 24911 10.84.127.254 wmi_helper windows_credentials Processing found ip addresses (non-snmp) for 10.84.127.254 notice 0.000000 60536345 9863 128756 2020-09-29 06:22:22 6 info 24911 10.84.127.254 m_devices_components nmap_ip Inserting ip 10.84.127.254 notice 0.000000 60536346 9863 128756 2020-09-29 06:22:22 7 debug 24911 10.84.127.254 wmi_helper windows_credentials Processing Nmap ports for 10.84.127.254 notice 0.000000 60536347 9863 128756 2020-09-29 06:22:22 7 debug 24911 10.84.127.254 m_rules execute Running rules::execute function. Device ID Input (update). notice 0.000000 Device ID supplied: 128756 60536348 9863 128756 2020-09-29 06:22:22 7 debug 24911 10.84.127.254 m_rules execute Completed rules::execute function. notice 0.000000 60536349 9863 128756 2020-09-29 06:22:22 5 debug 24911 10.84.127.254 wmi_helper windows_credentials No valid credentials for 10.84.127.254 fail 0.000000 60536350 9863 128756 2020-09-29 06:22:22 7 debug 24911 10.84.127.254 wmi_helper windows_credentials At IP 10.84.127.254, discovery found a device of type 'unknown'. notice 0.000000 60536351 9863 128756 2020-09-29 06:22:22 7 debug 24911 10.84.127.254 m_rules execute Running rules::execute function. Device ID Input (update). notice 0.000000 Device ID supplied: 128756 60536352 9863 128756 2020-09-29 06:22:22 7 debug 24911 10.84.127.254 m_rules execute Completed rules::execute function. notice 0.000000 60536353 9863 128756 2020-09-29 06:22:31 7 debug 24911 10.84.127.254 wmi_helper windows_credentials IP Audit finish on device 10.84.127.254 Peak Memory device complete 202.496848 118.941 MiB
- Mark Unwin
So it looks like the discovery is running and completing, but not updating that fact.
What's the time difference between discovery start and the last discovery log?
What happens on a /24 discovery?
Can you confirm the version of Open-AudIT you're running?
I upgraded to the latest version 3.4.
and
this newer version (3.3)
Mark.
- Jason
If it was completing, would it not show this as the last line, like a successfully finished discovery?
55037557 9848 NULL 2020-09-23 11:21:23 7 debug 23885 127.0.0.1 discoveries_helper ip_audit Discovery has finished. finished 1382.215097
Time from start to last entry in the discovery log is ~10 hours 7 minutes
59104748 9863 NULL 2020-09-28 20:15:12 6 info 19298 127.0.0.1 discoveries_helper discover_subnet Starting discovery for VLAN 96 start 0.000000 . . . 60536353 9863 128756 2020-09-29 06:22:31 7 debug 24911 10.84.127.254 wmi_helper windows_credentials IP Audit finish on device 10.84.127.254 Peak Memory device complete 202.496848 118.941 MiB
It also still shows as running.
Here is the same detail from a successful /24
Discovery detail
id name org_id description type devices_assigned_to_org devices_assigned_to_location network_address system_id other options discard last_run last_finished duration status ip_all_count ip_responding_count ip_scanned_count ip_discovered_count ip_audited_count edited_by edited_date 9848 VLAN 39 1 Subnet - 10.85.9.0/24 subnet 1 1 https://127.0.0.1/open-audit/ 0 {"nmap":{},"match":{},"subnet":"10.85.9.0\\/24"} n2020-09-23 10:40:01 2020-09-23 11:21:23 00:41:22 complete 256 201 201 172 41 Administrator 2020-03-10 13:20:45Last few log lines
55037464 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 m_rules execute Hit on $device->snmp_oid 1.3.6.1.4.1.11.2.3.2.6 eq 1.3.6.1.4.1.11.2.3.2.6 Rules Match - SNMP OID for Itanium notice 0.000470 {"model":"Itanium","type":"switch"} 55037465 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 m_rules execute Hit on $device->mac_address 00:18:fe:28:e0:6e st 00:18:fe Rules Match - Mac Address for Hewlett Packard notice 0.002763 {"model":"Itanium","type":"switch","manufacturer":"Hewlett Packard"} 55037466 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 m_rules execute Completed rules::execute function. notice 0.002763 55037467 9848 36952 2020-09-23 11:16:07 5 debug 13398 10.85.9.76 wmi_helper windows_credentials No valid credentials for 10.85.9.76 fail 0.000000 55037468 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 wmi_helper windows_credentials At IP 10.85.9.76, discovery found an unknown device. notice 0.000000 55037469 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 m_rules execute Running rules::execute function. Device ID Input (update). notice 0.000000 Device ID supplied: 36952 55037470 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 m_rules execute Hit on $device->snmp_oid 1.3.6.1.4.1.11.2.3.2.6 eq 1.3.6.1.4.1.11.2.3.2.6 Rules Match - SNMP OID for Itanium notice 0.000030 {"model":"Itanium","type":"switch"} 55037471 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 m_rules execute Completed rules::execute function. notice 0.000030 55037472 9848 36952 2020-09-23 11:16:07 7 debug 13398 10.85.9.76 wmi_helper windows_credentials IP Audit finish on device 10.85.9.76 Peak Memory device complete 1367.767546 82.993 MiB 55037557 9848 NULL 2020-09-23 11:21:23 7 debug 23885 127.0.0.1 discoveries_helper ip_audit Discovery has finished. finished 1382.215097I'm running the latest version, 3.4.0
Thank you for looking
- Mark Unwin
I'd suggest splitting the discovery into smaller chunks. As small as realistic. Personally I try to use a single /24 per discovery.
You can also try (depending on server specification, network performance, et al) increasing the queue limit. Default is 20. That means 20 IPs (after an initial 'ping') will be discovered at once. If you have the resources, try increasing it. All things being equal, it should process the IPs faster and may help.
Ideally your discoveries should complete within an hour.
- Jason
Hello,
I finally got around to working on this again. I deleted all my discoveries in preparation of importing a CSV of them all broken down in /24 networks.
However now I'm finding import does not work on this version (3.5.0) No matter what I try, I get this error message:
Permission denied for OrgID.
I am logged in as admin.
I've even tried a simple CSV formatted as such, but it fails.
name,org_id,type,network_address,other.subnet 192.168.1.0/24,1,subnet,http://127.0.0.1/open-audit/,192.168.1.0/24
Looking in the system log, I see this error:
User not authorised to use Org (discoveries:sub_resource_create).
Followed by this error
User not permittied to perform sub_resource_create on discoveries ID discoveries
- Mark Unwin
I'm /afk for Christmas. Will check when I return.
- Mark Unwin
This looks to be a bug. But it hasn't been triggered until now.
The POST url is to /discoveries/discoveries/import. It should be to discoveries/import. Because of that, the new response helper is assuming you have posted an ID (of "discoveries"), and the incorrect action is being chosen.
Fortunately it's an easy fix.
Edit the file open-audit/code_igniter/application/views/theme-bootstrap/v_collection_import. Replace the line
<form action="<?php echo $this->response->meta->collection; ?>/import" method="post" enctype="multipart/form-data">
with
<form action="import" method="post" enctype="multipart/form-data">
And it should work.
Also, make sure you enclose your CSV fields in double quotes, as per the import HTML page.
Mark.
- Jason
Thanks, I'll try it out this week.
- Jason
That worked for import. Thanks!
- Jason
I'm back to the problem where a discovery doesn't appear to do anything.
Any advice what to look at?
[root@openaudit ~]# mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM queue;" +------+----------+--------+--------+-----+--------+------------------------------------------------------+---------------------+---------------------+ | id | name | type | org_id | pid | status | details | edited_date | started_date | +------+----------+--------+--------+-----+--------+------------------------------------------------------+---------------------+---------------------+ | 1524 | VLAN 348 | subnet | 1 | 0 | queued | {"name":"VLAN 348","org_id":"1","discovery_id":9921} | 2021-01-06 09:50:00 | 2000-01-01 00:00:00 | +------+----------+--------+--------+-----+--------+------------------------------------------------------+---------------------+---------------------+[root@openaudit ~]# mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM discoveries WHERE id = 9921;" +------+----------+--------+------------------------+--------+-------------------------+------------------------------+-----------------+-----------+--------------------------------------------------+---------+---------+---------------------+---------------------+----------+---------+--------------+---------------------+------------------+---------------------+------------------+---------------+---------------------+ | id | name | org_id | description | type | devices_assigned_to_org | devices_assigned_to_location | network_address | system_id | other | options | discard | last_run | last_finished | duration | status | ip_all_count | ip_responding_count | ip_scanned_count | ip_discovered_count | ip_audited_count | edited_by | edited_date | +------+----------+--------+------------------------+--------+-------------------------+------------------------------+-----------------+-----------+--------------------------------------------------+---------+---------+---------------------+---------------------+----------+---------+--------------+---------------------+------------------+---------------------+------------------+---------------+---------------------+ | 9921 | VLAN 348 | 1 | Subnet - 10.84.48.0/21 | subnet | 1 | 1 | | 0 | {"nmap":{},"match":{},"subnet":"10.84.48.0\/21"} | | n | 2021-01-06 09:50:00 | 2021-01-06 09:50:01 | 00:22:09 | running | 0 | 0 | 0 | 0 | 0 | Administrator | 2021-01-06 09:48:21 | +------+----------+--------+------------------------+--------+-------------------------+------------------------------+-----------------+-----------+--------------------------------------------------+---------+---------+---------------------+---------------------+----------+---------+--------------+---------------------+------------------+---------------------+------------------+---------------+---------------------+[root@openaudit ~]# mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM discovery_log WHERE discovery_id = 9921;"
- Mark Unwin
The queue can take a minute to start - it's not necessarily instant. It should start though.
Maybe check cron and apache (access and error) logs?
What happens if you curl http://localhost/open-audit/index.php/util/queue ?
- Jason
It failed after 22 minutes.
[root@openaudit ~]# mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM discoveries WHERE id = 9921;"
+------+----------+--------+------------------------+--------+-------------------------+------------------------------+-----------------+-----------+--------------------------------------------------+---------+---------+---------------------+---------------------+----------+--------+--------------+---------------------+------------------+---------------------+------------------+---------------+---------------------+
| id | name | org_id | description | type | devices_assigned_to_org | devices_assigned_to_location | network_address | system_id | other | options | discard | last_run | last_finished | duration | status | ip_all_count | ip_responding_count | ip_scanned_count | ip_discovered_count | ip_audited_count | edited_by | edited_date |
+------+----------+--------+------------------------+--------+-------------------------+------------------------------+-----------------+-----------+--------------------------------------------------+---------+---------+---------------------+---------------------+----------+--------+--------------+---------------------+------------------+---------------------+------------------+---------------+---------------------+
| 9921 | VLAN 348 | 1 | Subnet - 10.84.48.0/21 | subnet | 1 | 1 | | 0 | {"nmap":{},"match":{},"subnet":"10.84.48.0\/21"} | | n | 2021-01-06 09:50:00 | 2021-01-06 09:50:01 | 00:22:09 | failed | 0 | 0 | 0 | 0 | 0 | Administrator | 2021-01-06 09:48:21 |
+------+----------+--------+------------------------+--------+-------------------------+------------------------------+-----------------+-----------+--------------------------------------------------+---------+---------+---------------------+---------------------+----------+--------+--------------+---------------------+------------------+---------------------+------------------+---------------+---------------------+apache log is full of items like this:
127.0.0.1 - - [06/Jan/2021:17:37:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:37:29 -0600] "GET /open-audit/index.php/widgets HTTP/1.1" 200 41512 "-" "Mojolicious (Perl)" 127.0.0.1 - - [06/Jan/2021:17:38:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:39:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:40:03 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:41:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:41:33 -0600] "GET /open-audit/index.php/widgets HTTP/1.1" 200 41512 "-" "Mojolicious (Perl)" 127.0.0.1 - - [06/Jan/2021:17:42:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:43:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:44:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:45:04 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:46:02 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:47:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:47:39 -0600] "GET /open-audit/index.php/widgets HTTP/1.1" 200 41512 "-" "Mojolicious (Perl)" 127.0.0.1 - - [06/Jan/2021:17:48:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:49:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:50:02 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:51:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:52:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:53:02 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:54:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:55:08 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:55:14 -0600] "GET /open-audit/index.php/util/queue HTTP/1.1" 200 - "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:56:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0" 127.0.0.1 - - [06/Jan/2021:17:57:01 -0600] "POST /omk/open-audit/tasks/execute HTTP/1.1" 200 2 "-" "curl/7.72.0"
cron log full of this
Jan 6 17:55:08 openaudit CROND[22651]: (root) CMD ( /usr/local/omk/bin/open-audit_tasks.sh >/dev/null 2>&1) Jan 6 17:56:01 openaudit CROND[22888]: (root) CMD ( /usr/local/omk/bin/open-audit_tasks.sh >/dev/null 2>&1) Jan 6 17:57:01 openaudit CROND[22981]: (root) CMD ( /usr/local/omk/bin/open-audit_tasks.sh >/dev/null 2>&1) Jan 6 17:58:01 openaudit CROND[23066]: (root) CMD ( /usr/local/omk/bin/open-audit_tasks.sh >/dev/null 2>&1) Jan 6 17:59:01 openaudit CROND[23183]: (root) CMD ( /usr/local/omk/bin/open-audit_tasks.sh >/dev/null 2>&1)
I started the discovery again and waited a while but nothing happened.
Running curl http://localhost/open-audit/index.php/util/queue does not result in anything happening.
- Mark Unwin
It failed after 22 minutes.
Sounds right, if no logs are received for 20 minutes AND the last log is not "Discovery has finished", then it's classed as failed.
Have you allowed http from localhost? This is a requirement, see - Configuring Open-Audit with HTTPS/SSL
Is there anything in the apache error_log?
What do you get if you query as below?
mysql -u openaudit -popenauditpassword openaudit -e "SELECT * FROM discovery_log WHERE discovery_id = 9921;"
Mark.
- Jason
yes, http is enabled and validated works via curl.
The discovery_log is empty.
- Mark Unwin
Is there anything in the apache error_log?
- Jason
Hello,
No, there was nothing in the error log.
I've found that once openaudit hangs on a few discoveries and if it has enough of these in a failed/forever running state, nothing will discover any more when you start it. And even though it shows several discoveries "running" the queue is empty.
If I delete ALL discoveries, then re-import them from csv as new ones, discoveries will start up again, however eventually one or more of them fail/run forever.
- Mark Unwin
If you're using the Pro/Ent GUI, there is an Advanced section where you can delete existing queue items.
Also make sure you're running the latest release. There have been some discovery bugs addressed.
Add your comment...
Hello,
Is re-running a subnet discovery supposed to update the details for devices found in that subnet? For example software, uptime, OS version, last seen on attribute, etc? I'm not seeing these update on subsequent runs of the discovery.
Thank you