On the system, there should be a "SWLEMAgent.lax" file inside the agent install directory (regardless of platform). Within this LAX file are some configuration parameters. You're looking for the "lax.nl.java.option.additional" parameter and you need to add "-Xmx" and "-Xms" arguments that specify the memory settings (ms = minimum memory, mx = maximum memory; best for high throughput is to allocate enough memory at startup, possibly the same number). The argument should already be there with other info, so be careful to not mess with what's there.
It should look something like this, you just want to add the bits at the end (bold - sets agent to use 256MB) and restart the service:
lax.nl.java.option.additional=-Djava.library.path=5.3.1\\lib -Xms256m -Xmx256m
Since the default is 64/128M, we usually start at 256M and work our way up. I've seen high throughput windows event log systems at 512M and we allocate up to multiple gigs to the appliances (when we sold hardware syslog servers, they had 12G and we'd allocate 5G to the agent... oh my!).