1. Start Registry Editor (Regedt32.exe).
2. Locate and then click the following key in the registry:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Memory Management
3. On the Edit menu, click Add Value, and then add the following registry value:
Value name: PoolUsageMaximum
Data type: REG_DWORD
Radix: Decimal
Value data: 60
Setting the value at 60 informs the Memory Manager to start the trimming process at 60 percent of PagedPoolMax rather than the default setting of 80 percent. If a threshold of 60 percent isn't enough to handle spikes in activity, reduce this setting to 50 percent or 40 percent.
Value name: PagedPoolSize
Data type: REG_DWORD
Radix: Hex
Value data: 0xFFFFFFFF
Setting PagedPoolSize to 0xFFFFFFFF allocates the maximum paged pool in lieu of other resources to the computer.
Caution
The 0xFFFFFFFF PagedPoolSize setting is not recommended for use on 32-bit Windows Server 2003-based computers that have 64GB of RAM. This will potentially bring the Free System PTE entry down and can cause continuous reboot of the computer. For this configuration, carefully choose a value based on the requirements and available resources.
4. Quit Registry Editor.
5. Restart the server for the changes to take effect.
Source: unable to allocate memory - Windows Server | Microsoft Docs