+
1 7 7
  1. #1
    Status
    Offline
      Rise Company
    Engineering and Technology
    Apr 2014
    Egypt
    4,611
    10

    FortiGate High memory usage Conserve Mode


    FortiGate High memory usage
    Steps on how to optimize Memory consumption - load did not decrease
    FortiOS 6.2: memory issues - Memory optimization techniques for FortiOS
    memory leak - Conserve Mode - ipsengine - IPS sensors are eating
    fortigate has too little RAM - Troubleshooting high memory usage
    Troubleshooting on high memory or high CPU usage
    Conserve mode - 80% memory ?



    1- What is your topology and traffic load?

    get system status
    get system session-info full-stat
    get system session-info statistics

    2- Can you provide the output of the following commands? Which process is using up the memory?

    get sys perf stat
    get sys perf top
    diagnose sys top
    diagnose sys top-summary "-s mem"

    3- Get more ips debug if it's confirmed the process is IPS

    diagnose ips memory status
    diagnose ips session list by-mem 10
    diagnose ips session status
    diagnose ips packet status

    4- Any relevant crash log?

    diagnose debug crashlog read

    5- Does this command fix your issue?

    diagnose test application wad 99

    Press ctrl + c to stop the "sys perf" report.

    ------------------------------------------------------------
    :
    ------------------------------------------------------------


    M p


    :
    get system performance top


    :
     - diagnose sys top 
    - diagnose sys top-summary 
    - diagnose test application ipsmonitor
    - diagnose test application ipsmonitor 99 
    - diagnose sys kill 11

    FortiGate Firewall Memory Usage goes high

    :
    https://kb.fortinet.com/kb/documentL...rnalID=FD33103
    https://kb.fortinet.com/kb/viewAttac...mentID=FD35192
    https://kb.fortinet.com/kb/documentL...rnalID=FD45766
    https://kb.fortinet.com/kb/documentL...rnalID=FD45932
    https://kb.fortinet.com/kb/documentL...rnalID=FD35126
    https://forum.fortinet.com/tm.aspx?m=173916
    https://forum.fortinet.com/tm.aspx?m=94366
    https://kb.fortinet.com/kb/documentL...rnalID=FD46971
    https://packetplant.com/wad-high-mem...oubleshooting/


    ------------------------------------------------------------------------
    Rise Company for Engineering & Technology
    ------------------------------------------------------------------------
    Web Hosting | Web Designing | E-Marketing

    # 1 Business Services

    Web Hosting - Business Emails

    Web Design - Google Adwords

    www.rise.company | www.rise.company/emails

    :
    ! .



  2. #2
    Status
    Offline
      Rise Company
    Engineering and Technology
    Apr 2014
    Egypt
    4,611
    10

    : FortiGate High memory usage Conserve Mode

    This is by no means a fix, but a work-around is to have the fgt perform a daily reboot.

    :
    config system global
    set daily-restart enable
    set restart-time <time value>
    end
    ------------------------------------------------------------------------
    Rise Company for Engineering & Technology
    ------------------------------------------------------------------------
    Web Hosting | Web Designing | E-Marketing

    # 1 Business Services

    Web Hosting - Business Emails

    Web Design - Google Adwords

    www.rise.company | www.rise.company/emails

    :
    ! .



  3. #3
    Status
    Offline
      Rise Company
    Engineering and Technology
    Apr 2014
    Egypt
    4,611
    10

    : FortiGate High memory usage Conserve Mode

    1- What is your topology and traffic load?

    get system status
    get system session-info full-stat
    get system session-info statistics

    2- Can you provide the output of the following commands? Which process is using up the memory?

    get sys perf stat
    get sys perf top
    diagnose sys top
    diagnose sys top-summary "-s mem"

    3- Get more ips debug if it's confirmed the process is IPS

    diagnose ips memory status
    diagnose ips session list by-mem 10
    diagnose ips session status
    diagnose ips packet status

    4- Any relevant crash log?
    diagnose debug crashlog read

    5- Does this command fix your issue?
    diagnose test application wad 99
    ------------------------------------------------------------------------
    Rise Company for Engineering & Technology
    ------------------------------------------------------------------------
    Web Hosting | Web Designing | E-Marketing

    # 1 Business Services

    Web Hosting - Business Emails

    Web Design - Google Adwords

    www.rise.company | www.rise.company/emails

    :
    ! .



  4. #4
    Status
    Offline
      Rise Company
    Engineering and Technology
    Apr 2014
    Egypt
    4,611
    10

    : FortiGate High memory usage Conserve Mode

    We have a case open with support for the conserve mode issue. We were running 6.0.x and they upgraded the box to 6.2.3. Still had issues. Their latest attempt to resolve it was to switch the box from proxy mode to flow mode for UTM. We've always used proxy mode, so I'm not sure what all that is going to impact. I have to check with my tech that is working on that client to see if it has made the problem go away or not.
    ------------------------------------------------------------------------
    Rise Company for Engineering & Technology
    ------------------------------------------------------------------------
    Web Hosting | Web Designing | E-Marketing

    # 1 Business Services

    Web Hosting - Business Emails

    Web Design - Google Adwords

    www.rise.company | www.rise.company/emails

    :
    ! .



  5. #5
    Status
    Offline
      Rise Company
    Engineering and Technology
    Apr 2014
    Egypt
    4,611
    10

    : FortiGate High memory usage Conserve Mode

    this seems to keep it under control for me. even on 6.2.1 they'll creep to 70% depending on what is enabled.
    This resets IPS every 6 hours and keeps mem around 55-60%, which I can live with.
    :
    config system auto-script
    edit "IPSReset"
    set interval 21600
    set repeat 0
    set start auto
    set script "diagnose test application ipsmonitor 99"
    next
    end
    --------------------------------------------------------------------------------------------

    Check if there is new software available. Usually, memory leak issues are quickly traced down and fixed with the next minor upgrade.
    Read the Release Notes! If you are running the latest avaiable software the main branch, there are 2 options:



    1. downgrade - I try to avoid it, as it's a messy solution
    2. schedule auto-restart of a process


    I'll focus on the second solution. In many cases, you can use it untill a new software version is released.

    I'll write a simple script that is executed every 12 hours:


    :
    config system auto-script
      edit restart_ipsmonitor
        set interval 43200
        set repeat 356
        set start auto
        set script 'diag test app wad 99'
      next
    end
    That script will automatically, every 12 hours, restart a wad process. Simple, but effective.
    Remember to remove it after a software upgrade to the verison which resolves this bug.

    ------------------------------------------------------------------------
    Rise Company for Engineering & Technology
    ------------------------------------------------------------------------
    Web Hosting | Web Designing | E-Marketing

    # 1 Business Services

    Web Hosting - Business Emails

    Web Design - Google Adwords

    www.rise.company | www.rise.company/emails

    :
    ! .



  6. #6
    Status
    Offline
      Rise Company
    Engineering and Technology
    Apr 2014
    Egypt
    4,611
    10

    : FortiGate High memory usage Conserve Mode

    About the diagnose sys top command

    You can use the diagnose sys top command from the FortiOS CLI to list the processes running on your FortiGate unit.
    The command also displays information about each process.

    Example output:

    CLI# diagnose sys top
    Run Time: 13 days, 13 hours and 58 minutes
    0U, 0S, 98I; 123T, 25F, 32KF
    newcli 903 R 0.5 5.5
    sshd 901 S 0.5 4.0
    Where the codes displayed on the second output line mean the following:

    * U is % of user space applications using CPU. In the example, 0U means 0% of the user space applications are using CPU.
    * S is % of system processes (or kernel processes) using CPU. In the example, 0S means 0% of the system processes are using the CPU.
    * I is % of idle CPU. In the example, 98I means the CPU is 98% idle.
    * T is the total FortiOS system memory in Mb. In the example, 123T means there are 123 Mb of system memory.
    * F is free memory in Mb. In the example, 25F means there is 25 Mb of free memory.
    * KF is the total shared memory pages used. In the example, 32KF means the system is using 32 shared memory pages.

    Each additional line of the command output displays information for each of the processes running on the FortiGate unit.
    For example, the third line of the output is:

    newcli 903 R 0.5 5.5
    Where:

    * newcli is the process name. Other process names can include ipsengine, sshd, cmdbsrv, httpsd, scanunitd, and miglogd.
    * 903 is the process ID. The process ID can be any number.
    * R is the state that the process is running in. The process state can be:
    o R running.
    o S sleep.
    o Z zombie.
    o D disk sleep.
    * 0.5 is the amount of CPU that the process is using. CPU usage can range from 0.0 for a process that is sleeping to higher values for a process that is taking a lot of CPU time.
    * 5.5 is the amount of memory that the process is using. Memory usage can range from 0.1 to 5.5 and higher.

    Interactive diagnose sys top commands

    You can enter the following single-key commands when diagnose sys top is running.

    * Press q to quit.
    * Press c to sort the processes by the amount of CPU that the processes are using.
    * Press m to sort the processes by the amount of memory that the processes are using.
    ------------------------------------------------------------------------
    Rise Company for Engineering & Technology
    ------------------------------------------------------------------------
    Web Hosting | Web Designing | E-Marketing

    # 1 Business Services

    Web Hosting - Business Emails

    Web Design - Google Adwords

    www.rise.company | www.rise.company/emails

    :
    ! .



  7. #7
    Status
    Offline
      Rise Company
    Engineering and Technology
    Apr 2014
    Egypt
    4,611
    10

    : FortiGate High memory usage Conserve Mode

    Fix

    Check if there is new software available. Usually, memory leak issues are quickly traced down and fixed with the next minor upgrade. Read the Release Notes! If you are running the latest avaiable software the main branch, there are 2 options:

    1. downgrade - I try to avoid it, as it's a messy solution
    2. schedule auto-restart of a process

    I'll focus on the second solution. In many cases, you can use it untill a new software version is released.
    I'll write a simple script that is executed every 12 hours:
    config system auto-script
    edit restart_wad
    set interval 43200
    set repeat 356
    set start auto
    set script 'diag test app wad 99'
    next
    endThat script will automatically, every 12 hours, restart a wad process. Simple, but effective. Remember to remove it after a software upgrade to the verison which resolves this bug.
    ------------------------------------------------------------------------
    Rise Company for Engineering & Technology
    ------------------------------------------------------------------------
    Web Hosting | Web Designing | E-Marketing

    # 1 Business Services

    Web Hosting - Business Emails

    Web Design - Google Adwords

    www.rise.company | www.rise.company/emails

    :
    ! .



  1. : 0
    : 22-12-2019, 04:50
  2. FortiGate Firewall Wad
    Rise Company Fortigate
    : 0
    : 29-08-2019, 06:08
  3. FortiGate Firewall ipsmonitor
    Rise Company Fortigate
    : 0
    : 29-08-2019, 03:48
  4. : 0
    : 29-08-2019, 03:13
  5. : 0
    : 29-06-2019, 23:21