المساعد الشخصي الرقمي

مشاهدة النسخة كاملة : فورتي جيت FortiGate استهلاك الرام High memory usage و Conserve Mode



Rise Company
01-05-2020, 07:34
فورتي جيت FortiGate حل مشكلة استهلاك الرام High memory usage
Steps on how to optimize Memory consumption - load did not decrease
FortiOS 6.2: memory issues - Memory optimization techniques for FortiOS
memory leak - Conserve Mode - ipsengine - IPS sensors are eating
fortigate has too little RAM - Troubleshooting high memory usage
Troubleshooting on high memory or high CPU usage
Conserve mode - 80% memory ?

https://www.rise.company/upload/uploads/158831150324141.png

1- What is your topology and traffic load?

get system status
get system session-info full-stat
get system session-info statistics

2- Can you provide the output of the following commands? Which process is using up the memory?

get sys perf stat
get sys perf top
diagnose sys top
diagnose sys top-summary "-s mem"

3- Get more ips debug if it's confirmed the process is IPS

diagnose ips memory status
diagnose ips session list by-mem 10
diagnose ips session status
diagnose ips packet status

4- Any relevant crash log?

diagnose debug crashlog read

5- Does this command fix your issue?

diagnose test application wad 99

Press ctrl + c to stop the "sys perf" report.

------------------------------------------------------------
شاهد الفيديو :
------------------------------------------------------------
https://www.youtube.com/watch?v=X2nIFwl8Hm8

استخدم هذا الامر ثم اضغط M للميمورى او p للبروسيسور


get system performance top

الاوامر التى تم استخدامها فى الفيديو



- diagnose sys top
- diagnose sys top-summary
- diagnose test application ipsmonitor
- diagnose test application ipsmonitor 99
- diagnose sys kill 11

شاهد ايضا
فايروول فورتي جيت FortiGate Firewall ارتفاع استهلاك الرام Memory Usage goes high (https://www.rise.company/forum/threads/46018-%D9%81%D8%A7%D9%8A%D8%B1%D9%88%D9%88%D9%84-%D9%81%D9%88%D8%B1%D8%AA%D9%8A-%D8%AC%D9%8A%D8%AA-FortiGate-Firewall-%D8%A7%D8%B1%D8%AA%D9%81%D8%A7%D8%B9-%D8%A7%D8%B3%D8%AA%D9%87%D9%84%D8%A7%D9%83-%D8%A7%D9%84%D8%B1%D8%A7%D9%85-Memory-Usage-goes-high)

المرجع:
https://kb.fortinet.com/kb/documentLink.do?externalID=FD33103
https://kb.fortinet.com/kb/viewAttachment.do?attachID=FD35192_MemoryUsageInsi ghtsInFortiOS5_0.pdf&documentID=FD35192
https://kb.fortinet.com/kb/documentLink.do?externalID=FD45766
https://kb.fortinet.com/kb/documentLink.do?externalID=FD45932
https://kb.fortinet.com/kb/documentLink.do?externalID=FD35126
https://forum.fortinet.com/tm.aspx?m=173916
https://forum.fortinet.com/tm.aspx?m=94366
https://kb.fortinet.com/kb/documentLink.do?externalID=FD46971
https://packetplant.com/wad-high-memory-usage-conserve-mode-and-general-high-cpu-memory-troubleshooting/

Rise Company
01-05-2020, 08:13
This is by no means a fix, but a work-around is to have the fgt perform a daily reboot.


config system global
set daily-restart enable
set restart-time <time value>
end

Rise Company
01-05-2020, 08:14
1- What is your topology and traffic load?

get system status
get system session-info full-stat
get system session-info statistics

2- Can you provide the output of the following commands? Which process is using up the memory?

get sys perf stat
get sys perf top
diagnose sys top
diagnose sys top-summary "-s mem"

3- Get more ips debug if it's confirmed the process is IPS

diagnose ips memory status
diagnose ips session list by-mem 10
diagnose ips session status
diagnose ips packet status

4- Any relevant crash log?
diagnose debug crashlog read

5- Does this command fix your issue?
diagnose test application wad 99

Rise Company
01-05-2020, 08:23
We have a case open with support for the conserve mode issue. We were running 6.0.x and they upgraded the box to 6.2.3. Still had issues. Their latest attempt to resolve it was to switch the box from proxy mode to flow mode for UTM. We've always used proxy mode, so I'm not sure what all that is going to impact. I have to check with my tech that is working on that client to see if it has made the problem go away or not.

Rise Company
01-05-2020, 08:32
this seems to keep it under control for me. even on 6.2.1 they'll creep to 70% depending on what is enabled.
This resets IPS every 6 hours and keeps mem around 55-60%, which I can live with.


config system auto-script
edit "IPSReset"
set interval 21600
set repeat 0
set start auto
set script "diagnose test application ipsmonitor 99"
next
end

--------------------------------------------------------------------------------------------

Check if there is new software available. Usually, memory leak issues are quickly traced down and fixed with the next minor upgrade.
Read the Release Notes! If you are running the latest avaiable software the main branch, there are 2 options:




downgrade - I try to avoid it, as it's a messy solution
schedule auto-restart of a process


I'll focus on the second solution. In many cases, you can use it untill a new software version is released.

I'll write a simple script that is executed every 12 hours:



config system auto-script
edit restart_ipsmonitor
set interval 43200
set repeat 356
set start auto
set script 'diag test app wad 99'
next
endThat script will automatically, every 12 hours, restart a wad process. Simple, but effective.
Remember to remove it after a software upgrade to the verison which resolves this bug.

Rise Company
01-05-2020, 08:43
About the diagnose sys top command

You can use the diagnose sys top command from the FortiOS CLI to list the processes running on your FortiGate unit.
The command also displays information about each process.

Example output:

CLI# diagnose sys top


Run Time: 13 days, 13 hours and 58 minutes
0U, 0S, 98I; 123T, 25F, 32KF
newcli 903 R 0.5 5.5
sshd 901 S 0.5 4.0

Where the codes displayed on the second output line mean the following:

* U is % of user space applications using CPU. In the example, 0U means 0% of the user space applications are using CPU.
* S is % of system processes (or kernel processes) using CPU. In the example, 0S means 0% of the system processes are using the CPU.
* I is % of idle CPU. In the example, 98I means the CPU is 98% idle.
* T is the total FortiOS system memory in Mb. In the example, 123T means there are 123 Mb of system memory.
* F is free memory in Mb. In the example, 25F means there is 25 Mb of free memory.
* KF is the total shared memory pages used. In the example, 32KF means the system is using 32 shared memory pages.

Each additional line of the command output displays information for each of the processes running on the FortiGate unit.
For example, the third line of the output is:


newcli 903 R 0.5 5.5

Where:

* newcli is the process name. Other process names can include ipsengine, sshd, cmdbsrv, httpsd, scanunitd, and miglogd.
* 903 is the process ID. The process ID can be any number.
* R is the state that the process is running in. The process state can be:
o R running.
o S sleep.
o Z zombie.
o D disk sleep.
* 0.5 is the amount of CPU that the process is using. CPU usage can range from 0.0 for a process that is sleeping to higher values for a process that is taking a lot of CPU time.
* 5.5 is the amount of memory that the process is using. Memory usage can range from 0.1 to 5.5 and higher.

Interactive diagnose sys top commands

You can enter the following single-key commands when diagnose sys top is running.

* Press q to quit.
* Press c to sort the processes by the amount of CPU that the processes are using.
* Press m to sort the processes by the amount of memory that the processes are using.

Rise Company
01-05-2020, 08:53
FixCheck if there is new software available. Usually, memory leak issues are quickly traced down and fixed with the next minor upgrade. Read the Release Notes! If you are running the latest avaiable software the main branch, there are 2 options:


downgrade - I try to avoid it, as it's a messy solution
schedule auto-restart of a process

I'll focus on the second solution. In many cases, you can use it untill a new software version is released.
I'll write a simple script that is executed every 12 hours:
config system auto-script
edit restart_wad
set interval 43200
set repeat 356
set start auto
set script 'diag test app wad 99'
next
endThat script will automatically, every 12 hours, restart a wad process. Simple, but effective. Remember to remove it after a software upgrade to the verison which resolves this bug.