Ffmpeg: Describing the audio data manually

I'm trying to manually create the packets from the audio that will be sent to the ffmpeg encoder. I'm trying to do this in the s16le format.

I'm to write the audio after I've written the video frame like this:

// Write a frame to the ffmpeg stream
fwrite(frame, sizeof(unsigned char*) * frameWidth * frameHeight, 1, ffmpeg); 

// Write multiple sound samples per written frame
for (int t = 0; t < 44100/24; ++t)
    fwrite(&audio, sizeof(short int), 1, ffmpeg);

I think that this should be possible but I don't know how to describe the audio without directly using the libavformat library that does this automatically.

Is this possible? If so, how do I describe the audio properly? I'm currently getting integer values from a sine wave.