Potential XE3 gotcha – dodgy old code vs. new TStream overloads

In XE3, TStream has acquired a number of new helper methods for Read and Write. Mostly these take the form of strongly-typed versions of Read, Write, ReadBuffer and WriteBuffer, called ReadData, WriteData, ReadBufferData and WriteBufferData respectively. These relieve you of the need to use SizeOf:

var
  Flag: Boolean;
  MagicNum: Integer;
begin
  Stream.ReadBufferData(Flag);
  if Flag then Stream.ReadBufferData(MagicNum);

I think adding these methods was a small but good idea, and a much better one than the TBinaryReader/TBinaryWriter classes that were pointlessly added in XE (cf. IOUtils, TStringBuilder, IEnumerable… ever get the feeling someone on the development team just never got over Big Bad EMBT Management killing off Delphi.NET?). It’s annoying whoever wrote the code hasn’t heard of static arrays, but the interface is fine.

However, there are also new overloads for Read, Write, ReadBuffer and WriteBuffer themselves. These take a TBytes parameter rather than an untyped buffer, and are presumably done for compatibility with the already-announced new compiler for targeting iOS and Android:

  function Read(var Buffer: TBytes; Count: Longint): Longint; overload;
  function Read(var Buffer: TBytes; Offset, Count: Longint): Longint; overload;
  function Write(const Buffer: TBytes; Count: Longint): Longint; overload;
  function Write(const Buffer: TBytes; Offset, Count: Longint): Longint; overload;
  procedure ReadBuffer(var Buffer: TBytes; Count: Longint); overload;
  procedure ReadBuffer(var Buffer: TBytes; Offset, Count: Longint); overload;
  procedure WriteBuffer(Buffer: TBytes; Count: Longint); overload;
  procedure WriteBuffer(Buffer: TBytes; Offset, Count: Longint); overload;

As shown by this QC report, the new overloads (and in particular, the ones without an additional Offset parameter) will alas be problematic though if you have old code that writes pointers on a stream on the understanding the addresses should be written.

Given pointer addresses are transient, this is in principle an odd thing to do. For myself, failing to dereference a pointer when calling Read- or WriteBuffer has caused infuriating crashes on more than one occasion! However, if you needed a list of integers in the past and lazily used a raw classic TList for the task, casting between Pointer and Integer as necessary, the code will need to be rewritten. The following, for example, will not work in XE3:

uses
  System.SysUtils, System.Classes;

var
  I: Integer;
  IntList: TList;
  Stream: TMemoryStream;
begin
  Stream := nil;
  IntList := TList.Create;
  try
    IntList.Add(Pointer(11));
    IntList.Add(Pointer(22));
    IntList.Add(Pointer(33));
    Stream := TMemoryStream.Create;
    //save list to the stream
    for I := 0 to IntList.Count - 1 do
      Stream.WriteBuffer(IntList.List[I], SizeOf(Integer));
    //clear the list before adding the items back from the stream
    IntList.Clear;
    Stream.Position := 0;
    while Stream.Read(I, SizeOf(Integer)) = SizeOf(Integer) do
      IntList.Add(Pointer(I));
    //show 'em
    for I := 0 to IntList.Count - 1 do
      WriteLn(Integer(IntList[I]));
  finally
    IntList.Free;
    Stream.Free;
  end;
end.

The problem here is that the when a value typed to Pointer is passed to an overloaded method that takes either an untyped paramater or a dynamic array, the dynamic array will be picked. This then causes an access violation, since the pointer being passed here isn’t actually a dynamic array of Byte – it’s an integer hard cast to a Pointer.

Now, perhaps there shouldn’t be type assignability here (for myself, I’d prefer only being able to assign a typed pointer of the appropriate sub-type, e.g. PByte to TBytes, PString to dynamic array of string and so on), however this has always been the case IIRC ever since dynamic arrays were added in D4. Morevoer, the proper way to do things before generics (which in practical terms meant until the generic TList actually worked properly) would have been to encapsulate TList, or at the very least hide the Pointer access members with Integer ones, a la Contnrs.pas’ TObjectList.

Because of these two points I think the closing of the QC report correct, notwithstanding the fact the narrative is wacko (‘test case error’? Er, no. XE3 has ‘fixed’ earlier behaviour? Er, no again – both old and new behaviours work correctly, and both stem from identical compiler semantics, they just conflict in a particular sort of case). Nonetheless, the overloads are still something to be wary of when recompiling old code.