Unable to update linux devices after 3.3.2 update

 
1
0
-1

We upgraded to 3.3.2 back in May. The server started up and displays data fine - everything appeared to be working. It took several weeks for me to notice that the weekly audit scripts (we run audit scripts manually against our linux environment using Rundeck) were failing.

Took a good chunk of time to narrow down what is happening, and I'm still not 100% sure where it's breaking down. I thought the issue was the submission from the audited systems, but at this point, even if I use the manual import and copy paste the contents of an xml file, I still wind up getting a 503 Service Unavailable error.

I've checked the php logs, the apache logs, and I've even enabled logging in codeigniter. Nothing is pointing to the source of the problem. It just tries to parse the xml file and seems to time out and die.

I'm kind of at my wits end trying to find where this is failing. Nothing shows up in any of the logs beyond the 503 message in the apache access log.

xml files being generated by the audit_linux.sh script on Red Hat 7 systems. Any help would be appreciated.

    CommentAdd your comment...

    3 answers

    1.  
      1
      0
      -1

      I confirmed that data= is not at the beginning of the xml files.

      Of note, I did try manually pushing one to the server using curl from the command line directly and got a response starting with "Disallowed Key Characters in" - this suggested some xml validation issues. I commented out the regex match in MY_Input as a temporary troubleshooting step to get by this.

      I can provide one of the xml files privately if there's a means here, but I can't share it publicly obviously. If this is a matter of a linux process being identified that's producing problem text, I can't find it. The XML files pass XML validation when I've tested them.

      I also don't want to get bogged down on the submission process. These fail even if I take the contents of an XML file and submit it manually via text inside the open-audit app itself.

      Output from the requested command - (  bracketed text replaces actual server names, etc below...):


      [root@[SERVER] tmp]# ./audit_linux.sh submit_online=y create_file=n url=http://[Actual.URL.of.Server]/open-audit/index.php/input/devices debugging=5
      Starting audit
      ----------------------------
      Open-AudIT Linux audit script
      Version: 3.3.2
      ----------------------------
      My PID is 6105
      Audit Start Time 2020-07-24 14:00:28
      Create File n
      Submit Online y
      Debugging Level 5
      Discovery ID
      Org Id
      Script Name audit_linux.sh
      URL http://[Actual.URL.of.Server]/open-audit/index.php/input/devices
      File /var/tmp/[SERVER]-20200724140028.xml
      ----------------------------
      System Info
      Policy Info
      BIOS Info
      Processor Info
      Memory Info
      Motherboard Info
      Optical Drives Info
      Video Cards Info
      Sound Cards Info
      Shares Info
      Network Cards Info
      Hard Disk Info
      Guest (Docker, Proxmox, LXC) Info
      Docker Info
      Log Info
      Environment Variable Info
      Swap Info
      User Info
      Group Info
      Software Info
      Service Info
      systemd services
      init.d services
      Server Info
      find: [REMOVED] : No such file or directory
      Route Info
      Netstat Info
      Custom Files Info
      Audit Generated in '48' seconds.
      Submitting results to server using cURL
      URL: http://[Actual.URL.of.Server]/open-audit/index.php/input/devices
      % Total % Received % Xferd Average Speed Time Time Time Current
      Dload Upload Total Spent Left Speed
      0 294k 100 252 0 0 35328 0 --:--:-- --:--:-- --:--:-- 50400
      <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
      <html><head>
      <title>302 Found</title>
      </head><body>
      <h1>Found</h1>
      <p>The document has moved <a href="https://[Actual.URL.of.Server]open-audit/index.php/input/devices">here</a>.</p>
      </body></html>
      Audit Completed in '49' seconds.

      1. Mark Unwin

        Your server is responding to https, but you're using http in your command line arguments. Change that.

      2. Alex Szele

        That explains the 302. That was because I copy-pasted the command you requested that I run. Everywhere else, we use https. I just didn't notice that your request had http in it. Now I get the 503 that I originally reported.

        [root@[SERVER] tmp]# ./audit_linux.sh submit_online=y create_file=n url=https://[Actual.URL.of.Server]/open-audit/index.php/input/devices debugging=5
        Starting audit
        ----------------------------
        Open-AudIT Linux audit script
        Version: 3.3.2
        ----------------------------
        My PID is 18530
        Audit Start Time 2020-07-27 08:44:32
        Create File n
        Submit Online y
        Debugging Level 5
        Discovery ID
        Org Id
        Script Name audit_linux.sh
        URL https://[Actual.URL.of.Server]/open-audit/index.php/input/devices
        File /var/tmp/[SERVER]-20200727084432.xml
        ----------------------------
        System Info
        Policy Info
        BIOS Info
        Processor Info
        Memory Info
        Motherboard Info
        Optical Drives Info
        Video Cards Info
        Sound Cards Info
        Shares Info
        Network Cards Info
        Hard Disk Info
        Guest (Docker, Proxmox, LXC) Info
        Docker Info
        Log Info
        Environment Variable Info
        Swap Info
        User Info
        Group Info
        Software Info
        Service Info
        systemd services
        init.d services
        Server Info
        find: [REMOVED]: No such file or directory
        Route Info
        Netstat Info
        Custom Files Info
        Audit Generated in '50' seconds.
        Submitting results to server using cURL
        URL: https://[Actual.URL.of.Server]/open-audit/index.php/input/devices
        % Total % Received % Xferd Average Speed Time Time Time Current
        Dload Upload Total Spent Left Speed
        100 294k 100 299 100 293k 11 11083 0:00:27 0:00:27 --:--:-- 0
        <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
        <html><head>
        <title>503 Service Unavailable</title>
        </head><body>
        <h1>Service Unavailable</h1>
        <p>The server is temporarily unable to service your
        request due to maintenance downtime or capacity
        problems. Please try again later.</p>
        </body></html>
        Audit Completed in '77' seconds.

      3. Mark Unwin

        I would check the Apache logs.

      4. Alex Szele

        Per my original post, I've reviewed php, apache, and codeigniter logs. All the apache log has is a single line 503 error on the page request. It shows that the post submission size matches the expected size of the xml file being submitted.

        The problem seems to be that open-audit is receiving the xml file and having some kind of issue processing it, but isn't logging an error anywhere. Is there anywhere to enable debug / verbose logging in open-audit beyond just the codeigniter logging?


      5. Mark Unwin

        Attach your result file to an email to marku@opmantek.com and I'll take a look.

      CommentAdd your comment...
    2.  
      1
      0
      -1

      The result of using create_file=y should remove the data= characters from the start of the file. Please check and advise if it is not.

        CommentAdd your comment...
      1.  
        1
        0
        -1

        I have just tested on Centos 6 and Redhat 7, both via discoveries and manually running it on the target machines. Both work as intended. Can you try running the below and posting the console output, please. Obviously replace YOUR_SERVER with your Open-AudIT servers resolvable name or IP.

        ./audit_linux.sh submit_online=y create_file=n url=http://YOUR_SERVER/open-audit/index.php/input/devices debugging=5

        Mark.

          CommentAdd your comment...