Recently I noticed errors on my Ubuntu 18.04 machine in /var/log/apache2/errors.log that look as below. These may also occur on any other Linux server running Apache and ModSecurity.
These would show multiple times for any requested page.
According to the Atomicorp ModSecurity page here - https://support.atomicorp.com/hc/en-us/articles/360000188468-Rule-execution-error-PCRE-limits-exceeded-8-null- you should increase a couple of limits.
I have edited /etc/modsecurity/modsecurity.conf and set these as recommended below.
I restarted Apache (sudo systemctl restart apache2) and I have no more warnings in my Apache error log.
With the release of Open-AudIT 3.2.0 comes a major new feature - Rules.
Rules as a collection of entries that essentially say "If the device has an attribute with X, then make the device's other attribute Y.". That may seem abstract, so what about "If the device has an SNMP OID of 220.127.116.11.18.104.22.168.620, then it's a Cisco 1851 router.".
Out of the box we have rules for MAC address prefixes, SNMP manufacturer IDs and all the aforementioned SNMP OIDs previously within Open-AudIT. We also ship various other rules that were previously hard coded. All up, for our first release we're shipping almost 100,000 rules!
"So what?" you say "What does this mean for me?" - well it means that you no longer need to send me your OIDs and device models, for a start. You can create custom Rules that will detect (almost) anything you like and set the appropriate device attribute.
The Rules are processed when a device's details are processed - during discovery and/or upon processing an audit result (hence, they usually run multiple times). Rules conform to the usual priority system - they will override every thing that's not a user input via the GUI. Rules are considered to be YOUR rules. Not something derived from a device. Hence they mean more than (say) something retrieved via SSH or SNMP or WMI. This is because if they don't do what you want YOU CAN CHANGE THEM.
NOTE - At present we cannot delete a rule input or output that contains a /. This is because the framework is parsing the / as part of the URL and returning a 404, even before our code runs. The work-around for this is to delete the Rule itslef, then recreate the inputs and outputs as required. Fortunately inputs and outputs that contain a / are rare (indeed, none exist by default).
Rules have two main sections - inputs and outputs.
Inputs are what is used to detect and match an attribute (or multiple attributes).
Outputs are what is to be set if the inputs match.
Inputs can use several operators to detect a match, not just equals. We can use the following operators:
does not equal
greater than or equals
less than or equals
like (which is case-insensitive)
not like (again, case insensitive)
in (a list)
not in (again, a list)
When we test multiple attributes in a single "input", those attributes are ANDed together. You cannot OR them. The below is an example (from the database, stored as JSON).
This translates to If system.manufacturer = Ubiquiti Networks Inc. AND system.sysDescr like UAP then we have a match.
The corresponding "output" section from this rules states (again, in JSON from the database):
Which means set the system.type to wap and the system.model to UniFi AP. Outputs can set an attribute using one of three 'types'. a number, text or a timestamp. For a timestamp you have the option of providing a date/time OR leaving it blank and having the system use the current date/time when the Rule is processed.
And don't worry - we wouldn't ask you to write the JSON. The web interface takes care of that for you. Of course, if you're using the API, then the JSON creation is on you :-)
One further item of note is the "weight" attribute we assign Rules. By default it's 100, but any Rules with a higher or lower weight will be run before or after those at weight 100. This provides a way to order the list in which Rules are applied. Mostly you won't need to worry about this, but if required, it's a life-saver.
Rule inputs also don't need to apply only to the "system" table. You can have a Rule on the service table to say if we detect a service named "My Service" then set the device description to "My Service Server" (for a bad example).
At this stage, Rule outputs can only set an attribute (or multiple attributes) on the system table. Custom fields are not supported right now (stay tuned).
The Rules engine is used by Community and available for editing in Professional and Enterprise.
So, what's the downside? Well running 100,000 rules several times does take its toll. I was however pleasantly surprised to see it takes less than 1 second each time all 100,000 Rules are processed. It does however mean more memory is consumed. In my testing it uses about 500MB. You shouldn't need to worry about increasing the PHP memory limit as we do this in code, but you will need to keep an eye on your server. Those users that process many devices AT ONCE may run into memory constraints. In general though, most users shouldn't notice any discernible difference. If you do, first thing to try is giving the server a memory bump. It is a database application after-all, so more memory and fast disk is always the answer :-)
UPDATE - With the release of 3.2.2, we no longer store ~100,000 rules in the database. This was fine on my test device (a core i7, 16GB memory and Samsung 860 NVMe), but in practice was causing customers servers to choke.
As per the Release Notes for 3.2.2 -
So, that was a ride... In testing our new Rules feature worked a treat. In practice, not so much. Most servers (ie, not mine) can't cope with loading the rule set, even if we break it down to smaller chunks, when processing multiple devices. What to do? What to do? Well we've taken a small step back. Rules still exist as a feature, and they still work a treat. But instead of inserting 100,000 Rules into the database, we've split them up into four distinct files and implemented them as code only. Hence, no loading all 100,000 Rules, decoding JSON and running them against a device. Now we just load the files and run the statements. Much, much faster and more memory efficient. No load on MySQL, and hence the CPU also drops. No populating a massive recordset and hence the memory drops. The not so good thing - these are no longer editable in the GUI. But it's not the end of the world. You can still make Rules as you see fit and they will be run after the "default" rules (those in code), hence you can override the "default" rules. So we don't lose much, but we gain a LOT of performance. We also added a few new Rules for Mac Models.
For those curious, the "new" files that replace the Rules are:
|/open-audit/code_igniter/application/helpers/mac_helper.php||Matches MAC addresses to manufacturers.|
|/open-audit/code_igniter/application/helpers/mac_model_helper.php||Matches Apple manufacturer codes to models (stored in system.manufacturer_code).|
|/open-audit/code_igniter/application/helpers/snmp_model_helper.php||Matches the device's SNMP OID to a model and type.|
|/open-audit/code_igniter/application/helpers/snmp_model_helper.php||Matches the devices's SNMP OID to the manufacturer.|
One final thing of note is the new GUI widget. Because we have almost 100,000 Rules, it's just not feasible to display them all in a list in the GUI. UPDATE - this is still in place, but you will not see all 100,000 Rules in the GUI as now (as per above) most are back in code files. So we don't. We have built a new widget that sits on the panel header and is used to search the Rules. Input anything and the rules name, description, inputs and outputs will be searched and anything matching will be returned. That result-set will still be limited to the default page size (1,000 items), so don't simply search for Cisco and expect to retrieve every Rule (there are 7,828 Cisco Rules by the way).
With this feature we essentially remove the "Tell us your unknown devices" issue as well as provide a powerful tool for you to automatically set attributes of your liking to your devices. Easy.
And one more item - we now have the ability to export a Rule - or anything else for that matter. Exporting an item will provide a JSON object that you can then use for Import. The export button is on each items details page and the import button is on the list page for each collection. With this in place, feel free to send us or post to Questions any Rules or Queries you think others may benefit from.
I often utilise Postman to query the Open-AudIT API when developing.
Just using a browser, it's difficult to send anything other than a GET request - but Postman makes it simple to send a POST, PATCH or DELETE as required.
You can get it from https://www.getpostman.com/downloads/ for Windows, Mac and Linux.
Install and start Postman. You can elect to create an account or not. You can also elect to create a new item using the wizard, or just close the modal and jump in. Let's do that
For the below, my Open-AudIT server is running on 192.168.84.4. You should substitute the IP address of your Open-AudIT server.
First you need to make a post to /login to get a cookie. Set the dropdown to POST and the URL to. Set the header Accept to application/json. Set the Body to form-data and provide the username and password keys, with values as appropriate for your installation. By default it will look as below. Now click the Send button.
You should see the JSON result saying you have been authenticated.
Once that's done, it's time to request some data. Make a GET request to and you should get a JSON response containing a list of devices. You can see the start of the JSON in the screenshot below.
What about changing the attribute of an item? Not too difficult. You'll need the ID of the device you want to change, along with the attribute name from the database. You can see these in the application by going to menu → Admin → Database → List Tables and clicking on the "system" table. Let's change the description for our device with ID 14.
You'll need to create a JSON object and assign it to the "data" item to do this. It's not too difficult. Your JSON object should look like below (formatted and indented for easy reading).
It looks worse than it is. Normally you would use code to do this, so it's a simple two line conversion. Because we're using Postman, we'll have to do it ourselves. A useful site is https://jsonlint.com/
So now you have your payload, let's send it to Open-AudIT. Make a new PATCH request and use the URL
With the release of Open-AudIT 3.1.0 we have massively expanded the options around keeping and processing data from devices.
SubSections of a device within Open-AudIT refers to the many tables that hold specific data types - software, netstat ports, processors, memory, disks, users, groups, etc, etc.
These options exist (for now at least) in the Configuration of Open-AudIT. The items of interest are create_change_log* . and delete_noncurrent*.
We previously had these options for a couple of select couple of Subsections, but have expanded these to cover every subsection.
Create Change Logs
The items named create_change_log_* use the database table names to specify which subsection they apply to - so create_change_log_software and create_change_log_memory are both valid examples. You can override ALL items by setting create_change_log to "n" - this will stop any change logs being generated, regardless of the individual table setting. So if a device has a piece of software added (for example), a correspond change log would not be inserted if create_change_log_software was set to "n". This is set to "y" by default. This matches how Open-AudIT has always worked.
We have also introduced three special configuration items for Netstat Ports. Because ports above 1024 are mostly designed to be dynamic, we now provide three options to keeping this data. create_change_log_netstat_registered, create_change_log_netstat_well_known and create_change_log_netstat_dynamic. These options correspond to the ports 0-1023, 1024-49151 and 49152-65535.See https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers. In particular, Windows DNS servers open a LOT of ports high in the range that are (in my opinion) silly to keep track of, see here and here. By default, only create_change_log_netstat_registered is set to "y". We may add to these options in the future for other subsection, if required.
Delete NonCurrent Items
Along similar lines, the configuration items for delete_noncurrent* use the database table names to specify which subsection they apply to. If set to "y", then no historical entries will be kept for that table, only the "current" items as at the last audit (or discovery). Again, these individual items can be overridden by the global "delete_noncurrent" item. If set to "y", it will remove all noncurrent items from all tables. This is set to "n" by default. This matches how Open-AudIT has always worked.
Hopefully these options provide some customisability for you to only keep the data you actually need.
Onwards and upwards.
After some extensive code rework, Open-AudIT 3.1.0 has been released for both Linux and Windows.
The Release Notes for Open-AudIT v3.1.0 provide specific details, but at a high level we've been changing the codebase to allow us to provide Open-AudIT Cloud in the near future. A hosted version of Open-AudIT that is always up to date, gets the latest code before regular releases, gets patches and bug fixies straight away and minimises the amount of infrastructure on premise required to get Open-AudIT up and running.
There have also been improvements around The Default Network Address, example device data, audit subsection processing, logging improvements, bug fixes and more.
I hope you find Open-AudIT as useful as I do.
With the new release of Open-AudIT 3.1.0, we no longer require the configuration item "default"network"address" to be set for Discoveries. It is still required for the "Audit My PC" functionality, but we hope to minimise this dependance going forward as well.
Why was Default Network Address required?
Initially when we ran a discovery, on both Linux and Windows, we ran the audit script in such a way that it needed to know where to submit it's result. What URL should it use - hence the requirement for the configuration item. A while back now we changed how Discoveries ran under Linux, removing this requirement.
Linux discoveries send the audit script to the target, run it with a flag of "submit_online = n" and "create_file = w". So do not submit the result to the server, create a file and output the filename to the console. The server waits for the script to finish and captures the console output. It now has the filename of the result on the target system. It copies the result from the target to itself and processes it. All good so far.
We could never make Windows work this way. The account we use for Apache is the standard "Local System" account. This account has no access to network resources. Hence it cannot simply copy the script to or from a target PC. This was always a pain because the Linux way of running the Discovery was so much better and cleaner. After some (more) research we realised we can use network resources via "net use" - we simply don't assign a drive letter. Yay! So Windows now can copy the audit script to the target, run it, wait for the console output and then copy the result file back and process it, just like Linux.
NOTE - If you are seeing issues copying the scripts when using the default "Local System" account, please change the Apache service to use another assount. This account does not need any special access (as credentials are supplied for devices in Open-AudIT itself), it just needs "network" access.
All that is a long explanation for "we don't need the default network address set". That's one less item a user needs to worry about.
We do still have the requirement to set the default network address for the functionality of the "Audit My PC" on the login page. We have plans to minimise this as well - if you can view the login page, we can use the request URL and work out what the default network address should be.
For now it's still required (as at 3.1.0), but look for it to be removed as a requirement in a near future release.
One step at a time, we're trying to make Open-AudIT as easy to use as possible.
Onwards and upwards.
After a long and interesting v2 series, we welcome v3 into the world!
Why v3 you ask? Well, with the recent improvements around discovery scan options and the resulting dramatic increase in discovery times along with the Windows version finally updating its XamppLite package to the latest full Xampp install, we thought it warranted the version increase.
We have introduced a new field for your devices called "identification" along with a new device type of "unclassified". This is used when we have some information about a device, but have not been able to talk to it using SNMP, WMI or SSH. So if we have a MAC address, or we know port X is open, then we know something and provide a type of unclassified. We populate the identification field with what we know. In the case where we literally have an IP and possibly a DNS hostname, the device will remain unknown in type. You can see these details on the Devices section of the Discoveries screen. This should help point you in the right direction to identifiying a device, rather than us throwing our hands in the air and saying "we couldn't talk to it, so it's unknown".
The new Discovery Scan Options are fully customisable for Enterprise users, choosable on a per discovery basis for Professional users and selectable on an install basis for Community users. We set the default to use UltraFast options and the increase in speed (especially on Linux servers) is, to put it midly, massive. We have genuine reports of a customer scanning their /22 and having the scan time drop from 29 hours to under 10 minutes. That is not a lie or exaggeration. I know it sounds hard to believe. Obviously there are surrounding conditions - network speed, device speed, reduced Nmap ports reporting, etc - but the result is genuine. To say we're happy with the performance is understating it quite a lot
We have finally managed to move Windows users from XamppLite to full Xampp. This wasn't without it's challenges, the largest being the PHP changes from 5.3 to 7.3 and encryption functions. In order to facilitate this, we use the existing 2.x install to export the encrypted credentials for credentials, device specific credentials, clouds and LDAP servers to a file, upgrade Xampp and Open-AudIT, then on the database schema upgrade, use the generated text file to encrypt the credentials and update the database entries. Once that's done and you're happy everything has worked, you can delete the file c:\xampplite\open-audit\migrate.json (we do not delete this file automatically). We also do not remove the old XamppLite installation. That is left for the user to decide to delete it at a time when they're happy with their new Open-AudIT 3.0.0 install.
Open-AudIT has never been easier to use, faster or more customisable than it is right now.
If you haven't upgraded, get on board!
As at Open-AudIT 2.3.2 and later, we have introduced some easy to use and extremely powerful options for discovering devices. These options centre around directing Nmap on how to discover devices.
We have grouped these options into what we're calling Discovery Scan Options. We ship seven different groups of options (items) by default that cover the common use-cases.
This benefits Community, Professional and Enterprise customers.
Feature availability is dependent on license type as per the table below.
|Match Rules - set default for all discoveries||y||y||y|
|Discovery Scan Options - set default for all discoveries||y||y||y|
|Discovery Scan Options - read||y||y|
|Discovery Scan Options - set per discovery||y||y|
|Discovery Scan Options - create, read, update, delete||y|
|Discovery Scan Options - Custom per Discovery||y|
|Discovery Scan Options - Exclude IP, range, subnet per discovery||y|
|Discovery Scan Options - Exclude ports per discovery||y|
|Discovery Scan Options - Set device timeout, per discovery||y|
|Discovery Scan Options - Custom SSH port per discovery||y|
|Match Rules - set per discovery||y|
Discovery Scan Types
The Discovery Scan Options we ship are detailed in the table below. As above, Enterprise users can create more of these or edit the shipped items.
|Approximate time in seconds for remote IP scan||1||5||40||90||100||240||1200|
|Must Respond to Ping||y||y||y||n||y||y||n|
|Use Service Version Detection||n||n||n||n||n||y||y|
|Consider Filtered Ports as Open||n||n||n||y||n||y||y|
|Top Nmap TCP Ports||10||100||1000||1000||1000||1000|
|Top Nmap UDP Ports||10||100||100||100||1000|
|Custom TCP Ports||22,135,62078||62078||62078||62078||62078||62078||62078|
|Custom UDP Ports||161||161|
|Exclude TCP Ports|
|Exclude UDP Ports|
|Timeout per Host|
|Exclude IP (address, range, subnet)|
|Custom SSH Port|
1The item for Medium (Classic) is similar to the Nmap for Discovery setting available in Open-AudIT 2.3.2.
Check the wiki here for a deeper look at Discovery Scan Options.
Example Scanning Improvement
We have a customer who is running discovery on a /22. The scan time to complete when using the original (hard set) options, prior to 2.3.2 was 29 hours. Using 2.3.2's UltraFast option, that scan now takes less than 10 minutes. To say they are impressed would be an understatement! They are now left with a smaller set of unknown devices that they can run a more detailed audit against. And remember, if the audited device is a computer, you will have a list of open ports derived from Netstat, anyway - possibly saving another audit cycle.
Handling Duplicate Serials
Recently we had cause to scan a subnet that was made up of virtual Cisco networking devices. These devices all happened to have identical serial numbers. Using the Match Rules per Discovery (available to Enterprise users) we were able to tweak the ruleset for this discovery only, without affecting other discoveries that rely upon matching a serial number. This ability solved a long-standing issue of working around a less than ideal setup on a network. A serial number, by definition, should be unique.
Networks respond differently depending on how they're configured. Some routers and/or firewalls can respond "on behalf" of IPs on the other side of their interfaces to the Open-AudIT Server. It is quite common to see Nmap report a probe for SNMP (UDP port 161) to respond as open|filtered for devices that do and do not exist. This is misleading as there is no device at that IP, yet it ends up with a device entry in the database. 99.9% of the time, it is not Open-AudIT, nor even Nmap, but the network causing this issue. Now that we have the options to treat open|filtered ports as either open or closed, we can eliminate a lot of this confusion. Enterprise users even have the option to change this on a per discovery basis (more than just using the Medium (Classic) item, as above).
Discovery Enterprise Options
The screenshot below is the Open-AudIT discovery page where all the audit configuration is set. I've added ample notes in the page explaining all the options making the tool easy to use for less technical staff.
Click to enlarge.
Check the wiki for a more detailed explanation about Discoveries
As well as the functional improvements to discovery, we have also revised the Discovery Details page. We have sections for Summary, Details, Devices, Logs and IP Addresses. The Devices section, in particular, is now much more useful. We have added a new type of Unclassified to the list and we use this when we have more than just an IP and/or name for the device. For instance, we may know it's IP, name and the fact that it has port 135 open. This at least is a good indication that the device is likely a Windows machine. So we know "something". More than just "there is something at this IP". That is now an Unclassified device. We still support Unknown devices as always - for those devices we really know nothing about. An example of this screen is below. We also provide a quick link to creating credentials when a service (SSH, WMI, SNMP) has been identified, but we were not able to authenticate to it.
We think these display improvements will go a long way to assisting you to remove any Unknown or Unclassified devices that are on your network.
Click to enlarge.
This new functionality makes Open-AudIT a powerful and easy to use discovery solution while providing great flexibility for advanced users.
I hope you enjoy the new features as much as our test customers and I do.
Your antivirus console should tell you which PCs have their antivirus software installed. But will it tell you which PCs don’t have their antivirus software installed? What about your server’s that live in a DMZ or another disconencted network? What about antivirus software from another vendor?
Because Open-AudIT captures the programs installed on a PC, Open-AudIT can report on specific installed programs very easily.
Open-AudIT contains a query for installed antivirus software which will tell you not only which PCs have which antivirus software installed, but also those without antivirus software installed.
Information is presented in an easily readable table format that is exportable to CSV (Excel), HTML, XML and JSON formats.
This is a very simple query and can easily be extended by the user to add additional software names when checking (if your antivirus software name doesn’t match the default names provided).
To enable the query go to menu -> Admin -> Queries -> Activate Query. You will see a list of available queries. Click the ‘tick’ icon on the right side to activate the “Installed Antivirus” query and make it appear in your menu’s.
Now go back to the homepage and click on the name of a group.
Once you see that group of devices, click menu -> Queries -> Installed – Antivirus.
Done. How easy was that!