IPSEC, Crypto and Ping: sendto: Out of memory

For Developers
1

I am trying to use an IPSec Tunnel between my router and my laptop using a crypto driver I'm developing.

"ping 10.0.0.2 -s1411" (or higher) works without any problem

any size lower will generate this error: "ping: send to: Out of memory"

I can use ping just fine in the other direction.

"ip -s x s" is showing the tunnel is up and packets are flowing.

Without installing my driver (using the generic hmac / sha256 and aes) and all else the same it works just fine. (no firewall or routing problems).

For debugging purposes I was dumping the raw-data that is processed and the driver is using the proper data for both encrypt and decrypt. I am using the extended test manager code and the driver is passing those tests just fine.

Where or What should I start looking at to debug this problem?

2

Getting encryption right is notoriously hard. I mean this with all respect, but if it’s just you developing this, or even you plus a small team, the odds are stacked against you in the solution be fully secure.

That’s why it is best to use one of the major encryption libraries, which are being used , tested, and developed by a huge number of people increasing the chances of security bugs being identified and fixed..

Can you provide more context on what your code is doing, that necessitates using it instead of something already standard?

3

I appreciate the fact that for a lot of mainstream uses, it's better to use fully test and developed libraries.

However since I started development of a crypto hardware driver I would like to see if I can (finally) finish it. To offload crypto to hardware should improve performance depending on the use case.

The question remains: Why would a ping with a larger size work? and why would a standard size work in the opposite direction given the driver does the same work. As far as its concerned doing an encrypt and later a decrypt or visa versa should matter.

4

I'm still not sure why / how I got those error messages, but after rebuilding from 5.4.31 to 5.4.33 it works okay now. (maybe something was wrong with my build, even it worked in software??)

Anyway: Context as to why:

5

Just as an update: I figured out which module is causing the error: esp4_offload

However using "only software" vs as hardware crypto solution, it actually enhances performance by 10-15% because of the GSO offloading. (using software as a fallback method doesn't help the problem, so it must be something else)... to be continued and debugged.