[cmucl-imp] string-to-octets

[Raymond Toy]

On 10/29/11 4:18 AM, Helmut Eller wrote:
> I'm wondering how STRING-TO-OCTETS can be used with a fixed sized byte
> buffer.  For example we want to write a long string to a byte stream
> using a fixed size byte buffer.  STRING-TO-OCTETS seems to return the
> buffer and the number of bytes written.  E.g.
>
> (let ((buffer (make-array 20 :element-type '(unsigned-byte 8)))
>       (string (make-string 100  :initial-element #\a)))
>   (stream:string-to-octets string :external-format :utf32 :buffer buffer))
>
> Returns a new vector (non-eq to buffer) and 404.
>
> Shouldn't one return value also indicate how many characters were
> converted so that the conversion can be continued at that character
> offset without allocating a fresh buffer?
>
Good question.  It seems that string-to-octets grew the buffer and the
new buffer has all of the converted characters and the original contains
just the that portion.

I'll have to look through the history to see why it is this way, but I
think you're right.  If a buffer is supplied, string-to-octets shouldn't
grow the buffer;  it should stop when the buffer is full.  Although, I
can see why it needs to:  the code doesn't know how many octets are
needed for a character until the character is converted, and by then we
may have exceeded the buffer, so the buffer gets a partially converted
character.

If you were going the other way (octets-to-string), there's
octets-to-string-counted that tells you how many octets were consumed
for each character.  This was needed to make streams fast because an
input buffer may not have a full character at the end of the buffer.

Ray