1- What is your topology and traffic load?

get system status
get system session-info full-stat
get system session-info statistics

2- Can you provide the output of the following commands? Which process is using up the memory?

get sys perf stat
get sys perf top
diagnose sys top
diagnose sys top-summary "-s mem"

3- Get more ips debug if it's confirmed the process is IPS

diagnose ips memory status
diagnose ips session list by-mem 10
diagnose ips session status
diagnose ips packet status

4- Any relevant crash log?
diagnose debug crashlog read

5- Does this command fix your issue?
diagnose test application wad 99